Systems, apparatus and methods for cost and performance-based management of resources in a cloud environment

ABSTRACT

Systems, methods and apparatus, including computer program products, are disclosed for regulating access of consumers (e.g., applications, containers, or VMs) to resources and services (e.g., storage). In one embodiment, this regulation occurs through the movement of consumers between different providers of a resource or service, such as a cloud service provider. Moving consumers includes, for example, determining the cost of moving the consumer from a first provider to a second provider. According to various embodiments, the cost of moving the consumer is compared to cost and performance criteria associated with moving the consumer from the first provider to the second provider. Cloud-based services may be priced as templates, reserved instances, or a combination.

FIELD

This specification relates generally to systems, apparatus and methodsfor managing resources in computer systems, including, but notexclusively, to the movement of computational demand among cloud serviceproviders based on the price of computer resources or computer resourcebundles available from multiple providers in the computer system.

BACKGROUND

Traditional computer system architectures typically include one or morededicated computer servers for each application being run, and are oftendesigned to include an excessive allocation of resources—for example,physical components such as central processing units (CPUs) andstorage—in order to ensure the ability to handle peak demands. Suchresource overloading can be costly, inefficient and difficult to scaleand manage.

So-called “cloud” providers offer various elements of computationalinfrastructure and processing capacity and services (e.g., applications,licenses, etc.) as a service via the internet. The term “cloud” connotesthe arbitrary location of the physical or software resourcescorresponding to the offered services; these are determined by the cloudprovider and may be altered to suit the provider's changing customerdemands and service-level commitments, all in a manner invisible tothose customers. In concept, cloud services are analogous tovirtualization, which refers to the abstraction of computer resourcesfrom their hardware or software-based physical constructs.Virtualization may be based on one or more virtual machines (VMs), eachof which is a software implementation of a computer that executesprograms or applications as if it were a physical computer. A virtualmachine operates like a physical computer and contains, for example, itsown virtual (e.g., software-based) CPU, random access memory (RAM), harddisk storage, and network interface card (NIC). Each virtual machine ina virtualization system generally runs its own guest operating system(OS), and the virtual machines generally share the underlying physicalmachine resources of the system. Virtualization generally involves theuse of physical resources belonging to a single entity and for whichthat entity is responsible, so virtualization facilitates more efficientuse of existing resources rather than “offloading” the expansion ofresources to an outside provider. VMs may be deployed on premises or inthe cloud.

There are many potential benefits to cloud services, most obviously theability to avoid capital investment in infrastructure whose utilizationcan vary. Often that variation is predictable, e.g., the pronouncedspike in resource utilization by retailers during the holiday season, inwhich case there is time for advance planning and comparison among cloudofferings. Even when demand fluctuations are not predictable, however,cloud resources can be added and, within contract restrictions,decommissioned on demand. Cloud services can also be used to improve theavailability and robustness of applications by shifting workloads to thecloud provider to handle fail-over situations, improve load balancing,and provide storage resiliency. Similarly, cloud computing providesflexible partitioning of applications, deployment, and operations.Notwithstanding the potential benefits, operating in a cloud environmentpresents various challenges and potential pitfalls, includingsignificant operations management challenges.

For example, different cloud providers bundle their servicesdifferently, offering packages involving different combinations ofresources for varying amounts of time. Evaluating and comparing theprices of these packages relative to a particular customer's needs canbe difficult and, today, often involves guesswork on the part ofinformation-technology managers. Maximizing economic efficiency meansnot only selecting the right package for current and anticipatedresource needs, but also making new selections as resource needs evolvealongside changing services offerings from alternative cloud providers.It may also involve predicting provider behavior based on, for example,past behavior (e.g., recurring or seasonal price adjustments). In acloud environment, the traditional “buy or build” decision becomes acomplex ongoing process of selection among competing service offerings.Cloud deployment also allows usage to be tracked conveniently andcontinuously, and cloud resource utilization can be trimmed or augmentedto match needs as they evolve.

Moving a set of cloud-based resources from one provider to anotherinvolves evaluating alternative providers for a service or resource bycomparing the attributes of the new service or resource and the currentone, again in the context of services that may be bundled differently bydifferent providers. Beyond direct cost considerations, moving largeamounts of data from one provider to another can involve indirect costsin terms of time and the involvement of the customer's own computationalresources.

In view of the foregoing, a need exists for an improved resourcemanagement system to address the administrative and technical challengesof cloud-based resource provision.

SUMMARY

Described herein are new technologies relating to the management ofresources and performance in cloud-based resource provisioningenvironments. For example, these technologies introduce the use ofsupply chain economics and other techniques to offer a unified platformto integrate, optimize or improve, and automate resource and performancemanagement in a cloud-based system. The economics-based methods can alsoextend to other systems for managing application performance.

Although the detailed description focuses primarily on cloud-basedprovisioning of infrastructure resources such as storage and CPUs, theapproaches described herein can also be applied to cloud-basedprovisioning of software (e.g., productivity suites, customerrelationship management software and human resources-managementsoftware), all of which the cloud vendor hosts and provides over theinternet; and so-called “platform as a service” offerings that providesoftware infrastructure, such as operating systems and middleware, viathe cloud. Moreover, infrastructure resources can also include networks,databases and other fundamental computing resources.

The systems, apparatus and methods contemplated and disclosed herein canbe used and applied based on, and in combination with, the disclosurescontained in the above-referenced patent applications. For example, theycan be applied to recommend and eventually migrate workloads amongmultiple providers. The systems, apparatus and methods depicted in theaccompanying drawings describe particular embodiments and are notintended to be exhaustive of the contemplated configurations andprocesses.

Accordingly, in a first aspect, the invention relates to acomputer-implemented method comprising, in various embodiments, thesteps of (a) determining, by a consumer manager running on a dataprocessor in a computer system, a cost in currency units for running acomputational workload on a first computational resource provider, theworkload having a resource requirement corresponding to a quantity ofone or more resources selected from CPUs, memory, databases, networkbandwidth, and/or input-output capacity for use by a component of thecomputer system; (b) selecting to run the workload on the firstcomputational resource provider; (c) determining, after a predeterminedperiod of time has passed since the selection or after the cost forrunning the workload has increased by at least a predetermined amount, acost for running the workload on a second computational resourceprovider, wherein the first computational resource provider is anin-house resource provider and the second computational resourceprovider is a cloud-based service provider offering a plurality oftemplates, each of the templates specifying a quantity of one or moreresources selected from CPUs, memory, databases, network bandwidth,and/or input-output capacity; (d) selecting, from among the plurality oftemplates, an optimal template (i) having resources exceeding theworkload resource requirement and (ii) having a lowest cost of alltemplates having resources exceeding the workload resource requirement;(e) determining a cost of moving the workload to the second providerusing the optimal template; (f) determining whether any templateresources exceeding the workload resource requirement can be deployed bythe consumer manager in the computer system; (g) computing a utilizationvalue for running the workload on the second provider based at least inpart on the determined cost of the optimal template on the secondprovider, the determined cost of moving the workload to the secondprovider, and whether any template resources exceeding the workloadresource requirement can be deployed by the consumer manager in thecomputer system; and (h) moving the workload to the second provider ifthe utilization value exceeds a utilization value of continuing to hostthe workload on the first provider.

In some embodiments, the computing step includes determining a ratio of(i) the sum of the determined costs for running the workload on thesecond provider and moving the workload to (ii) the determined remainingbudget capacity. The cost of running the workload on the second providermay be determined by computing the square of the ratio of 1 to (1-X),wherein X is determined by computing the ratio of (i) a sum of anestablished cost of the template and the budget spent by the templateover a predetermined period of time to (ii) a total budget available tothe workload over a second predetermined period of time.

In various embodiments, the method further comprises computing a secondutilization value for running the workload on a third provider based atleast in part on the determined cost for hosting the template on thethird provider, the determined cost of moving the workload to the thirdprovider, and a second determined remaining budget capacity, wherein thethird provider is another cloud-based service provider; and moving theworkload to the third provider based at least in part on the secondutilization value after the second utilization value has surpassed asecond predetermined value.

The cost of running the workload on the second provider may be based atleast in part on the utilization of one or more resources of the firstprovider and/or a determined performance characteristic of the first orsecond provider. The cost of running the workload on the second providermay be a dynamic, on-demand price based at least in part on one or moretemplate resources, or may be a reserved-instance price based at leastin part on multiple template resources.

In various embodiments, the method further comprises establishing theterms of continued running of the workload on the second provider for apredetermined length of time after the template has moved to the secondprovider. The determined cost for running the workload on the secondprovider may be based at least in part on an actual or anticipatedenvironmental impact, a contractual clause, a preferred vendor status, aquality of service (QoS) requirement, and/or a compliance or regulatoryrequirement; or may be based at least in part a cost of facilities, acapital amortization, and/or an operations cost. For example, the costfor running the workload on the second provider may increase as theremaining budget for the template decreases.

The remaining budget for running the workload may be adjusted based atleast in part on a determined service-level agreement (SLA) performancemetric. The method may include selecting a new cloud-based providerbased at least in part on the computed utilization value.

In another aspect, the invention pertains to a computer-implementedmethod comprising, in various embodiments, (a) determining, by aconsumer manager running on a data processor in a computer system, afirst cost in currency units for running a computational workload on afirst provider, wherein (i) the workload has a resource requirementcorresponding to a quantity of one or more resources selected from CPUs,memory, databases, network bandwidth, and/or input-output capacity, (ii)the first provider is a cloud-based service provider, and (iii) theworkload is hosted on the first provider as a first template specifyinga quantity of one or more resources selected from CPUs, memory,databases, network bandwidth, and/or input-output capacity, the firsttemplate resources exceeding the resource requirement; (b) determining,by the consumer manager, a second cost in currency units for running theworkload on a second provider as template different from the firstprovider, wherein (i) the second provider is a cloud-based serviceprovider and (ii) the second template specifies a quantity of one ormore resources selected from CPUs, memory, databases, network bandwidth,and/or input-output capacity, the first template resources exceeding theresource requirement; (c) selecting to run the workload on either thefirst or second provider based at least in part on a comparison of thedetermined costs; (d) determining a cost for running the workload on athird provider, wherein the third provider is an in-house serviceprovider; (e) determining a cost of moving the workload to the thirdprovider; (f) computing a utilization value for hosting the workload onthe third provider based at least in part on the determined cost forhosting the workload on the third provider and the determined cost ofmoving the workload to the third provider; and (g) moving the workloadto the third provider if the utilization value for running the workloadon the third provider exceeds a utilization value of continuing to runthe workload on the selected one of the first or second provider.

In some embodiments, the computing step includes determining a ratio of(i) the sum of the determined costs for running the workload on thesecond provider to (ii) a remaining budget. The cost of running theworkload on the third provider may be based at least in part on theutilization of one or more resources of the first or second provider. Itmay be dynamic, on-demand price based on one or more template resources.The cost of running the workload on the second provider may be areserved-instance price based at least in part on multiple templateresources. In some embodiments, the method further comprisesestablishing the terms of continued running of the workload on thefirst, second or third provider for a predetermined length of time.

In another aspect, the invention relates to a computer system formanaging allocation of workloads, the computer system comprising, invarious embodiments, a consumer manager configured to (a) determine acost in virtual currency units for running a computational workload on afirst computational resource provider, the workload having a resourcerequirement corresponding to a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity; (b) select to run the workload as a firsttemplate on the first computational resource provider, specifying aquantity of one or more resources selected from CPUs, memory, databases,network bandwidth, and/or input-output capacity, the first templateresources exceeding the resource requirement; (c) determine, after apredetermined period of time has passed since the selection or after thecost for running the workload has increased by at least a predeterminedamount, a cost for running the workload on a second computationalresource provider as a second template, wherein the second templatespecifies a quantity of one or more resources selected from CPUs,memory, databases, network bandwidth, and/or input-output capacity, thefirst template resources exceeding the resource requirement; (d)determine a cost of moving the workload to the second provider; (e)compute a cost of running the workload on the second provider based atleast in part on the determined cost of the second template and thedetermined cost of moving the workload to the second provider; and (f)cause the workload to be moved to the second provider if the cost ofrunning the workload on the second provider exceeds the cost of runningthe workload on the first provider.

In still another aspect, the invention pertains to acomputer-implemented method comprising, in various embodiments, thesteps of (a) determining, by a consumer manager running on a dataprocessor in a computer system, a set of resources for running aworkload required by a component of the computer system, the resourcesincluding a needed quantity of one or more of CPUs, memory, databases,network bandwidth, and/or input-output capacity; (b) identifying, by theconsumer manager, a first cloud-based resource offering by a firstprovider including the needed quantity of resources individually on anon-demand basis reflecting a utilization level of the resources anddetermining a cost for provisioning the needed quantity of resources tothe computer system on the on-demand basis; (c) identifying, by theconsumer manager, a second cloud-based resource offering by a secondprovider different from the first cloud-based resource offering andconsisting of a reserved instance, the reserved instance specifying aquantity of one or more resources selected from CPUs, memory, databases,network bandwidth, and/or input-output capacity fully utilized for afixed time period and corresponding to or exceeding the needed quantityof resources; (d) determining a cost charged for the reserved instance;and (e) selecting to utilize the reserved instance to run the workloadbased at least in part on a comparison of the determined cost forrunning the workload on the first provider given the utilization levelwith the determined cost of the reserved instance.

In various embodiments, the method further comprises the steps of (f)determining, after passage of a predetermined period of time no greaterthan the fixed time period associated with the reserved instance, a costfor running the workload on a third cloud service provider; (g)determining a cost of moving the workload to the third cloud serviceprovider; (h) computing a utilization value for running the workload onthe third provider based at least in part on the determined cost forrunning the workload on the third provider, the determined cost ofmoving the workload to the third provider, and a need by one or moreother components of the computer system for the remainder of thereserved instance; and (i) moving the workload to the third providerbased at least in part on the utilization value after the utilizationvalue has surpassed a predetermined value.

The first and second second cloud-based resource offerings may beoffered by the same or different cloud providers. The reserved instancemay exceed the needed quantity of resources and the computing stepaccounts for any restriction on moving an excess portion of the reservedinstance to another component of the computer system. The cost ofrunning the workload on the second provider may be based at least inpart on the utilization of one or more resources of the first providerand/or on a determined performance characteristic of the first or secondprovider. The cost of running the workload on the third provider may bea dynamic, on-demand price based on one or more characteristics of thecomputer system. The determined cost for running the workload on thesecond provider may be based at least in part on an actual oranticipated environmental impact, a contractual clause associated withthe reserved instance, a preferred vendor status, a quality of service(QoS) requirement, or a compliance and/or regulatory requirement. Thedetermined cost for running the workload on the second provider may bebased on a cost of facilities, a capital amortization, and/or anoperations cost. In some embodiments, the method further comprisesexchanging virtual currency units used for running the workload to agovernment-backed currency.

Yet another aspect of the invention comprises a computer-implementedmethod comprising, in various embodiments, (a) determining, by aconsumer manager running on a data processor in a computer system, a setof resources for running a workload required by a component of thecomputer system, the resources including a needed quantity of one ormore of CPUs, memory, databases, network bandwidth, and/or input-outputcapacity; (b) determining, by the consumer manager, a cost in currencyunits for running the workload on a first cloud service provider as afirst reserved instance specifying a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity fully utilized for a fixed time period, the firstreserved instance corresponding to or exceeding the needed quantity ofresources; (c) determining, by the consumer manager, a cost in currencyunits for running the workload on a second cloud service provider as asecond reserved instance specifying a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity fully utilized for a fixed time period, the secondreserved instance being different from the first reserved instance butcorresponding to or exceeding the needed quantity of resources; (d)selecting to run the workload on either the first or second reservedinstance based at least in part on a comparison of the determined costs;(e) determining a cost for running the workload on a third provider inthe computer system, wherein the third provider is an in-house provider,based on a historical resource utilization of the workload; (f)determining a cost of moving the workload to the third provider in thecomputer system; (g) computing a cost for running the workload on thethird provider based at least in part on the determined cost for runningthe workload on the third provider and the determined cost of moving theworkload to the third provider; and (h) moving the workload to the thirdprovider based on the computed cost.

The first cloud service provider may be the same as or different fromthe second cloud service provider. The selected reserved instance mayexceed the needed quantity of resources, in which case the computingstep accounts for any restriction on moving an excess portion of theselected reserved instance to another component of the computer system.The cost of running the workload on the third provider may be a dynamic,on-demand cost based on one or more characteristics of the computersystem.

In still another aspect, the invention pertains to a computer system formanaging allocation of workloads comprising, in various embodiments, aconsumer manager configured to (a) determine a set of resources forrunning a workload required by a component of the computer system, theresources including a needed quantity of one or more of CPUs, memory,databases, network bandwidth, and/or input-output capacity; (b) identifya first cloud service provider offering the needed quantity of resourcesindividually on an on-demand basis reflecting a utilization level of theresources and determining a cost charged by the first cloud serviceprovider for provisioning the needed quantity of resources to thecomputer system; (c) identify a second cloud service provider differentfrom the first cloud service provider and offering a first reservedinstance, the first reserved instance specifying a quantity of one ormore resources selected from CPUs, memory, databases, network bandwidth,and/or input-output capacity fully utilized for a fixed time period andcorresponding to or exceeding the needed quantity of resources; (d)determine a cost charged by the second cloud service provider for thereserved instance; (e) select to purchase the reserved instance to runthe workload on the second provider based at least in part on acomparison of the determined cost for running the workload on the firstprovider with the determined cost for running the workload on the secondprovider; (f) determine, after passage of a predetermined period of timeno greater than the fixed time period associated with the first reservedinstance, a cost for running the workload on a third cloud serviceprovider as a second reserved instance specifying a quantity of one ormore resources selected from CPUs, memory, databases, network bandwidth,and/or input-output capacity fully utilized for a fixed time period andexceeding the needed quantity of resources; (g) determine a cost ofmoving the workload to the third cloud service provider; (h) compute autilization value for running the workload on the third provider basedat least in part on the determined cost for running the workload on thethird provider, the determined cost of moving the workload to the thirdprovider, and a need by one or more other components of the computersystem for the resources of the second reserved instance exceeding theneeded quantity of resources; and (i) cause the workload to be moved tothe third provider based at least in part on the utilization value afterthe utilization value has surpassed a predetermined value.

In the ensuing discussion, cloud-based resources are acquired by a“consumer,” which may be a virtual (e.g., a VM), logical (e.g., anapplication), container, or physical (e.g., a computer) component. Theability of a consumer to obtain resources may be a function of itsrevenue and the expenses based on the price of the resources. In thescenario where a consumer is an entity that falls below its desiredservice level, the system can allocate additional budget to theconsumer, providing a greater power to buy resources. Therefore, QoSmetrics can be used to gain additional granularity in the allocation ofresources.

In other embodiments, the system can also be used when deploying newapplications in order to determine the required or preferred resourceallocation. By simulating application demand (e.g., transactions), thesystem can determine the required or preferred number of applicationcomponents and the appropriate allocation of resources.

Additionally and/or alternatively, the system can be used when planningfor future application needs. By simulating future application demand(e.g., transactions), the system can determine the required number ofapplication components and the appropriate allocation of resources.Moreover, particular embodiments hereof can be embodied in methods andsystems that regulate access of consumers to cloud-based resources andservices (e.g., memory, CPUs, storage).

Another aspect of the subject matter described herein can be embodied inmethods and systems that formulate and evaluate the option to move aworkload (demand) to a new cloud provider. According to variousembodiments, “formulating” can include the attributes taken into accountwhen considering the option to move to the new provider. The cost ofmoving can be part of the comparison between two or more alternatives(e.g., keeping a resource, such as memory, in an existinginfrastructure—which may involve physically expanding the resource toaccommodate increased utilization—or moving the resource to an externalcloud provider). It will be understood that reference to costs, pricesand the like herein refers to any measure of value, including but notlimited to suitable denominations or units of currency, such as virtualor electronic currency, physical currency, or electronic currency,whether or not tied to any government-issued or “real world” monetaryunit or system. For example, moving time can be expressed in a realvalue that quantifies the cost of machine downtime. In contrast, ifthere is a strict limit on acceptable downtime, the cost of moving aworkload to the cloud can be expressed in terms of a cost metric ofdowntime.

According to various embodiments, “evaluating” includes making thedecision (e.g., initiating an action based on the decision) anddetermining the right time to take the action. Compared to othereconomics-based decision-making systems, one embodiment described hereinpostpones the decision for the future, advantageously waiting asufficient amount of time until the decisionmaker is convinced that thedecision is the right one. Other embodiments of this aspect includecorresponding systems, apparatus, and computer program products.

In some embodiments, cloud resources are represented as “templates,”which represent a package of cloud-based resource offerings. A templatemay reflect the needs of a consumer but also specifies a set ofresources offered by one or more cloud providers for a known price. Thetemplate thereby permits cloud services to be generalized acrossproviders. Each template is associated with a cost for each cloudprovider that offers services corresponding to the template, eitherseparately (on demand) or as a bundle (which may be discounted relativeto the separate services). For example, a template may specify a bundleof storage and CPUs offered for a fixed period of time. In variousembodiments, templates are reviewed and assessed periodically asprovider offerings change and as consumer utilization patterns change tofavor different collections of resources.

In some embodiments, “evaluating” includes consideration of coupons or“reserved instances.” A reserved instance, like a template, specifies apackage of resources, but typically represents a promotion offered by acloud provider. Reserved instances may include temporal or otherrestrictions that are considered during evaluation. For example, systemsand methods in accordance herewith may monitor utilization and determinewhen a consumer should shift from on-demand provision of cloud servicesto a reserved instance. Moreover, utilization of reserved instances maybe monitored on an ongoing basis so that unused reserved instances,which have been paid for, can be shifted to other consumers.

Management of reserved instances is also relevant to virtualization.Container systems provide an operating-system level virtualization inwhich the kernel of an operating system can allow for multiple isolateduser space instances. Stated another way, a container is based on servervirtualization that uses a shared operating system. Rather thanvirtualizing hardware and creating whole virtual machines, each withtheir own operating systems, containers run atop the shared operatingsystem kernel and file system that looks and feels like a complete,isolated instance of the operating system. Like shipping containers forcargo, these software containers can ship applications across differentnetwork-based systems (e.g., cloud computing based systems) and limitthe impact of one container's activities on another container.

A container system may include software abstractions to virtualizecomputer resources (or compute resources) which are used by applicationsrunning in the container (“containerized” applications). The containersystem provides means to provision containers, allocate and control theresources available to a container, deploy and execute applications inthe container, and facilitate full use of the container resources bysuch containerized applications, while isolating them from otherapplications, sharing the underlying resources. When a containerizedapplication accesses a virtualized container resource (e.g., CPU,memory, storage I/O, Network I/O), the container system maps this accessto a direct access of the underlying real resource.

Additional details of one or more embodiments of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescriptions contained herein and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example container environment in whichresources are managed. [DF: don't we need to add the VM layer?]

FIG. 2 is a block diagram of an example software system for managingresources in a container system.

FIG. 3 is a flow diagram of an example process for using a platformmanager in a container system.

FIG. 4 is an example model for service provision and consumption in asupply chain container system.

FIG. 5 is a flow diagram of an example process for deploying a newconsumer element with a provider element in a container system.

FIG. 6 is a flow diagram of an example process for deliveringservice-level agreement targets through resource allocation in acontainer system.

FIG. 7 is a flow diagram of an example process for economically basedI/O scheduling in a container system.

FIG. 8A is an example purchase order data structure for use inpurchasing services from a provider element manager in a containersystem.

FIG. 8B is an example service confirmation data structure for use inconfirming or rejecting the purchase of services from a provider elementmanager in a container system.

FIG. 9 is an example process for managing the states of system elementsin a container system.

FIG. 10 is a block diagram of an example multi-domain software systemenvironment for managing virtualized resources.

FIG. 11 is a block diagram of an example virtualization environmentwhich illustrates supply chain relationships between service entitiesand resources.

FIG. 12 is a block diagram of another example virtualization environmentwhich illustrates supply chain relationships between service entitiesand resources in a container system.

FIG. 13 is a flow chart illustrating a process for resource scaling inthe virtualization environment of FIG. 11.

FIG. 14 is a flow chart illustrating a process for service entityscaling in the virtualization environment of FIG. 11.

FIG. 15 is a block diagram of yet another example virtualizationenvironment which illustrates the supply chain relationships betweenservice entities and resources in a virtualization environment and cancooperate with the processes described in FIGS. 13-14.

FIG. 16 is a block diagram of an example virtualization environment inwhich resources are managed by an action manager.

FIG. 17 is a block diagram illustrating the data flow for managingresources in the virtualization environment of FIG. 16.

FIG. 18 illustrates an exemplary block diagram of a virtualizationenvironment in which a virtual machine is determining whether to take anaction.

FIG. 19 illustrates another exemplary block diagram of a virtualizationenvironment in which a virtual machine is determining whether to migratebetween public and private service providers.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 illustrates a representative container system or environment 100in which resources are managed. The system 100 includes two servers 102,104 that run respective container systems 110, 112. The container system110, at the server 102, allocates computer resources (or computeresources) of the server 102—e.g., CPUs, memories, storage volume,storage, and/or network I/O pathways and bandwidth—to two containers120, 122. Similarly, the container system 112 at the server 104allocates resources of the server 104 to containers 124, 126. Thecontainers 120, 122, 124, 126 execute respective containerizedapplications 130, 132, 134, 136.

As previously discussed, container systems permit flexible organization.In the example system 100, the servers 102, 104 may be physical machineswith physical computer (or computational) resources. Alternatively, theserver 102 may be a virtual machine with virtualized resources while theserver 104 is a physical server. The containers 120, 122, 124, 126 maybe distinct containers, or replicated copies of a single container. Insome embodiments, a group of containers may be clustered into acontainer-Point-of-Delivery (cPOD) system, to run related applications.For example, a multi-tier Web service may include a containerized Webserver (shown as the application 130), a containerized applicationserver (shown as the application 134), and a containerized databaseserver (shown as the application 136). The Web server provided by theapplication 130 can maintain significant level of communications withthe application server provided by the application 134. The I/O pathwaybetween the applications 130, 134 traverses the application 130, thecontainer 120, the container system 110, an operating system 106, anetwork interface card (NIC) 140, a data network 160, a NIC 142, anoperating system 108, the container system 112, the container 124, andthe application 134.

In this example, the portion of the I/O pathway that includes the NIC140, the data network 160, and the NIC 142 traverses network switchesand links, and can thus result in significant I/O latency as well asbandwidth limitations. A container manager 236—considered below andshown, for example, in FIG. 2—can migrate (ship) the container 120, withthe application 130, from the server 102 to the server 104. Thismigration replaces the I/O pathway from the application 130 to theapplication 134 with a pathway that includes the application 130, thecontainer 120, the container system 112, the operating system 108, thecontainer system 112, the container 124, and the application 134.Advantageously, this modified I/O pathway entirely can be handled by theserver 104 through memory transfers. This in-memory I/O pathway cansupport very high memory transfers bandwidth and very low latency, thus,improving the cPOD performance.

Although a specific environment 100 including the two servers 102 and104 is shown in FIG. 1 and described above, it will be understood thatthe environment 100 is illustrative only. For example, the environment100 may include more than two servers, and each of the servers 102 and104 may be associated with any number of containers as desired. Theprinciples described herein may be applied regardless of the particularapplication or applications being run in the container system.

FIG. 2 shows a representative software system 200 for managing resourcesin container systems, such as the container system 100. According tovarious embodiments, the software system 200 may be used to allocateserver and I/O resources (such as CPU, memory, flash storage, hard drivestorage and I/O bandwidth) to containers. The software system 200 alsomay be used, for example, to monitor, detect, and handle congestionconditions at a resource (e.g., I/O pathway, memory, and so on) and tomove containers among available servers to optimize or improveapplication performance and resource utilization.

The software system 200 monitors, controls, and otherwise interacts withvarious managed container system elements (also referred to herein asservice elements or computer elements) through respectiveinstrumentation. As used herein, in the context of computers, the term“instrumentation” refers generally to any software and/or hardware thatprovides an ability to monitor, control, or otherwise interact with acomputer element, such as to detect operations events, reconfigureparameters, diagnose errors, and write trace information. For example,when a computer application contains instrumentation code, the computerapplication may be managed using a management tool.

Several example container system elements are shown in FIG. 2 as part ofan Information Technology (IT) Container Stack (ITCS) 202, includingapplications components 210, container systems 212, servers 214, storagesystems 216, networks 218, and operating resources 220 (such as powersupplies, cooling systems, and rack space). In some embodiments, theITCS 202 may include, for example, a proper subset or a proper supersetof these container system elements 210, 212, 214, 216, 218, 220.

As shown, the software system 200 includes a platform layer 230, whichprovides an infrastructure to manage, for example, the I/O flows in acontainer system (such as the example container system environment 100shown in FIG. 1). The platform layer 230 includes element managers 234,236, 238, 240, 242, 244. More particularly, the platform layer 230includes an application manager 234, a container system manager 236, aserver manager 238, a storage manager 240, a network manager 242, and anoperations manager 244. These element managers 234, 236, 238, 240, 242,244 use management instrumentation of respective elements to monitor andcontrol the respective elements of the ITCS 202.

For example, the server manager 238 may use built-in managementinstrumentation, such as Management Information Bases (MIBs) of theserver it is managing, to monitor the server's CPU, memory, and I/Ointerfaces (such as a Host Bus Adapter (HBA) and NICs) and to controltheir operational parameters. The server manager 238 may access suchmanagement instrumentation using standardized protocols (such as SimpleNetwork Management Protocol (SNMP)) or specialized mechanisms. In someembodiments, a proper superset or only a proper subset of these elementmanagers 234, 236, 238, 240, 242, 244 may be desired or needed incertain environments. For example, when the containers do not accessstorage, the use of a storage manager 240 may not be needed.Additionally, for example, an operating system element manager (notshown) may be included as part of platform layer 230.

As also shown, the platform layer 230 also includes one or more types ofmodeling databases 245. As discussed in more detail below, the databases245 may include supply chain modeling (SCM) databases 246 and operationsdatabases 248. The platform layer 230 also includes a platform manager250, which, as explained in greater detail below, can be responsible forgeneral provisioning, initializing, and management tasks.

The software system 200 shown in FIG. 2 also includes a functionalmanagement layer 252, which includes user interface (UI) software 260for use by administrators or other users to monitor and control acontainer system (such as the example container system environment 100shown in FIG. 1). For example, an administrator may use the UI software260 to set proactive automation policies to optimize or improveperformance and resource utilization, detect and resolve operationalproblems and performance bottlenecks, allocate priorities and usagecharges to different applications, and plan capacity expansions.

The functional management layer 252 also includes a collection offunctional managers 272, 274, 276, 278, which are used to enable usersto monitor, control, and automate the underlying automated managementmechanisms of container systems according to the principles describedherein. The software system 200 may alternatively include, for example,a proper subset or a proper superset of these functional managers.

As shown in FIG. 2, the functional management layer 252 includes anapplication manager 272, which, for example, enables users to select orconfigure respective parameters of a computer agent or process topartition application components among different containers, allocatesvirtual budgets to applications based on the business value of theirservices, as described in greater detail below, and specifies theresources required by the applications. The application manager 272 usesthe parameters to create respective records in the operations databases248. The platform manager 250 uses the operations records to initializerespective application managers 234, which use the operations records todeploy the applications 210, according to the principles describedbelow. Additional functions of monitoring and controlling applicationsmay be incorporated into the application manager 272.

The functional management layer 252 also includes a performance manager274, which allows users to monitor and control the delivery ofservice-level agreements (SLAs) to applications. For example, a user ofthe software system 200 can specify target SLA parameters—such aslatency or transaction rate—of one or more particular applications. TheSLA parameters are used by the software system 200 to adjust theperformance of the applications using the principles described below. Auser can also monitor the SLA parameters value, as well as therespective virtual payments made by an application, thereby correlatingthe application's budget with its SLA performance. Additional functionsof monitoring and controlling the performance of applications, as wellas the other elements of the ITCS 202, may be incorporated into theperformance manager 274.

A capacity manager 276 monitors relationships between the supply anddemand of resources in the ITCS 202. For example, the capacity manager276 may monitor the relationships over a predetermined time period,which can range from short term (such as a few minutes or one hour) tolong term (such as one day, week, month or year). In some embodiments,the capacity manager 276 maintains full accounting of revenues and costsand provides monitoring of these accounts and notifications upon certainaccounting events. The capacity manager 276, by itself or with theassistance of an incorporated or separate return-on-investment (ROI)manager (not shown), enables a user to monitor the ROI of the elementsin the ITCS 202. The ROI is defined as revenue divided by cost, whererevenue is the income from virtual payment collected by a selectedelement and cost is the virtual payments by the element for theresources that the element uses.

For example, a large ROI may indicate to the capacity manager 276 thatthere is excess demand over supply of the element capacity, and asustained high ROI may thus indicate insufficient capacity. The capacitymanager 276 compares a monitored ROI with specific and potentiallypredetermined ROI targets, which may be configured by an administratoror other user, to recommend capacity increases of particular elements tomeet demand. According to the supply chain economic principles describedbelow, the ROI of an element in the ITCS 202 may be considered as acentral metric of economic value.

The ROI may be calculated at any appropriate time and for anyappropriate duration over which revenue and cost are considered. Thus,the principles described herein provide an accounting framework toquantify and measure the value generated by components of the ITCS 202.For example, at the bottom of the ITCS 202, there are raw resources thatgenerate real (non-virtual) costs, such as monetary costs that are paidto an electric company. At the top of the ITCS 202, there areapplications that play roles in generating real (non-virtual) revenues,such as monetary sales revenue received from customers. It is possibleto treat one or more of the system elements 210, 212, 214, 216, 218, 220as virtual profit and loss (P&L) entities, generating revenues throughpayments by its consumers, and paying the costs of services it consumes.The use of virtual currency pricing and payments, as described herein,to distribute a share of these revenues to cover costs increases theefficiency and overall ROI of the entire system.

A back-charging manager 278 monitors and accounts for the virtual cashflows between elements in the ITCS 202 and enables users to flexiblycompute financial metrics of interest. For example, users can monitormetrics describing the allocation of application budgets to acquiresupply chain resources, the allocation of a resource among the workloadsof different applications, the ROI efficiency of different resources,and application budgets required to deliver particular SLAs. Thesemetrics and other parameters may be used to support policies onbudgeting applications, adjusting the budgets to represent changingprices, capacity, and demand of resources along the supply chain, andconverting virtual currency used within the software system 200 to realcurrency (such as United States dollars, or euros) that is generated bythe business units who own the applications and that may be used to payfor IT resources.

The platform manager 250 can manage a container system using anysuitable means described herein, including using a process 300 as shownin FIG. 3, which illustrates an example process 300 for using theplatform manager 250 in a container system (such as the container system100). According to various embodiments that implement process 300, theplatform manager 250 initializes, or launches, the functional managers272, 274, 276, 278 of the functional management layer 252 for a specificcontainer environment (step 302). The platform manager 250 discovers themanaged container system elements of the ITCS 202 in the containerenvironment (step 304). This discovery is handled, for example, throughstandard processes to get configuration data from the container system,OS, server, network, and storage systems.

The platform manager 250 also initializes, or launches, an elementmanager (such as one or more of element managers 234, 236, 238, 240,242, 244, described above) for each group of respective elements of agiven class of elements that have been discovered (step 306). Forexample, the platform manager 250 may detect a DELL server and a SUNserver, and the corresponding groups of respective elements may both beassigned respective element managers. The platform manager 250configures the element managers to monitor and control the respectiveelements via respective management instrumentation.

The platform manager 250 populates and initializes the platform modelingdatabases 245—for example, the supply chain modeling databases 246 andthe operational databases 248 (step 308)—and starts monitoring certainpotential changes of the managed environment (step 310). For example,the container system 100 may be monitored to determine if there havebeen any container changes, such as any added, deleted, or migratedcontainers (decision block 312). If a container change has beendetected, the platform manager 250 again initializes the elementmanagers as described above.

If no container changes have been detected, the presence of containersystems is evaluated to determine if there have been any containersystem changes, such as any added or deleted container system (decisionblock 314). If a container system change has been detected, the platformmanager 250 again discovers the managed container system elements of theITCS 202 in the container environment as described above. Otherwise, theplatform manager 250 evaluates whether there have been any major networkchanges (decision block 316), in which case the platform manager 250similarly re-discovers the managed container system elements of the ITCS202 in the container environment as described above. For example, theplatform manager 250 may discover loss or gain of network I/O pathways,congestion or under-utilization of an I/O pathway, low or excessivelatency of an I/O pathway, or packet losses along an I/O pathway.Otherwise, the platform manager 250 evaluates whether there have beenany major storage changes (decision block 318). For example, theplatform manager 250 may discover storage I/O congestion, or alternateI/O pathways that would provide better (i.e., lower) access latency. Ifmajor storage changes have been detected, the platform manager 250 againdiscovers the managed container system elements of the ITCS 202 in thecontainer environment as described above.

If no container, container system, network, or storage changes have beendetected, the platform manager 250 determines whether to continuemonitoring of the same (decision block 320). If the platform manager 250decides to continue monitoring, the platform manager 250 again startsthe monitoring of potential changes of the managed environment.Otherwise, the process 300 ends (end block 322).

The order of steps in the example process 300 described above is forillustration purposes only, and can be done in different orders. Forexample, the platform manager 250 may evaluate whether there have beenany major storage changes (decision block 318) before determiningwhether there has been any major network changes (decision block 316).Moreover, additional steps may be included, for example, to protect thesoftware system 200 against its own failures. Such additional steps mayinclude, for example, inserting between steps 308 and 310 describedabove the steps (not shown) of creating a mirror and backup copies ofthe platform image (including the databases 246 and 248), running asecond instance of the software system 200 in standby mode andmonitoring the primary instance of the software system 200, andswitching to the standby instance of the software system 200 upondetecting the failure of the first instance of the software system 200.

According to various embodiments, the software system 200 describedabove can operate using a supply chain software model of the ITCS 202that it manages. In other words, each container system element 210, 212,214, 216, 218, 220 of the ITCS 202 is modeled as a provider and aconsumer of services. For example, FIG. 4 shows an example model 400 forservice provision and consumption in a supply-chain containerenvironment. According to various embodiments as shown in FIG. 4, whichincludes references to the container system elements of the ITCS 202shown in FIG. 2, the server 214 may consume services of the operatingresources 220, including, for example, power 402, cooling 404, physicalspace 406, a share of capital expenditure (CAPEX) costs 408, and a shareof operating expenditure (OPEX) costs 410. The server 214 further mayconsume the resources of the networks 218, including, for example, alocal area network (LAN) 420 and a storage area network (SAN) 422.

However, the server 214 may provide the container systems 212 withvarious physical resource services, including, for example, CPUbandwidth 430, memory 432, network I/O bandwidth 434, and storage I/Obandwidth 436. The container systems 212 may also consume storageresources 438 from the storage element 216, and, in turn, may offerservices (such as services 440, 442) to the application 210. Theapplication 210, on the other hand, may offer services to respectivebusiness activities of one or more business units 450.

According to various embodiments, the allocation of resources and theprocessing of workloads through the supply chain, as described above,may be performed through the use of virtual currency. In these cases,supply chain elements use virtual currency to pay for the services theyconsume and to price the services they offer. For example, a selectedapplication 210 may receive a budget from its business users reflectingthe business value of the services that it offers. The application 210may shop for a container system 212 that offers the lowest pricedprocessing services that the application 210 requires, and may use itsvirtual budget to pay for these services. The container system 212, inturn, may use its income of virtual currency to pay for the servicesoffered by the server 214, the network 218, and the storage system 216.Each of the container systems elements 210, 212, 214, 216, 218, 220 ofthe ITCS 202 may price their services in virtual currency to reflecttheir costs, and additionally, or alternatively, to balance supply anddemand.

According to various embodiments, resource pricing may also be based oneor both of capacity or performance characteristics. For example, theserver 214 or a cloud provider may offer multiple types of processors orCPUs, each with respective clock rates and other characteristics, atdifferent prices. Similarly, for example, storage I/O resources in thestorage system 216 and network I/O resources in the network 218 orsupplied by a cloud provider may be priced according to their bandwidthand latency characteristics. This manner of pricing can take intoaccount that, as noted above, I/O pathways internal to a server (i.e.,interconnections of containers co-located with a single server, e.g.,the containers 120 and 122 as shown in FIG. 1) typically offer higherbandwidth and lower latency than I/O pathways between containers locatedat different and distinct servers (e.g., the containers 120 and 124 asshown in FIG. 1). Thus, for example, one or more of the components andresources associated with internal I/O pathways (or the aggregate ofsuch components and resources) may be priced lower than components andresources (alone or in the aggregate) for pathways traversing switchesand/or involving multiple servers. Alternatively, for example,components and resources associated with such internal I/O pathways maybe priced higher to account for an expected increase in performance andthus value to the acquiring entity. For decisions involving clouddeployment,

The supply chain model of the ITCS 202 is primarily maintained by thesupply chain model databases 246 shown in FIG. 2. According to variousembodiments, the supply chain model databases 246 may include one ormore financial databases to debit and credit the respective accounts ofcustomers and providers to reflect the transfer of virtual payments, asdiscussed in greater detail below. It will be understood, however, thatnon-monetary transactions may be entered into between a consumer and aprovider.

The supply chain model databases 246 may be object-relationshipdatabases, such that elements of the supply chain are modeled as objectscorresponding to services to be offered. As used herein, the term“objects” refers to data structures including data fields and methods.Examples of service objects include simple and composite serviceobjects. According to various embodiments, simple service objects—orobjects relating to the provision of a single type of service—mayinclude the following types of attributes:

<service-identifier, units, used, available, duration, price( )>.

The “service-identifier” attribute may itself include the followingtypes of attributes as descriptors of the service that may be used for aparticular class of services: <name, type, description, elementmanager>. For example, a CPU service provided by a Dell® server with anIntel iQ9550® processor managed by an element manager ServerEM015 may beassigned the following identifier: <Dell4, CPU, iQ9550, ServerEM015>.The “units” attribute may measure the quantity of service, such as 5 Mhz(CPU), 2 GB (memory) or 10 Mbps (net I/O). The “used” attribute mayrefer to the amount of the service or resource capacity that is alreadycommitted. The “available” attribute may refer to the amount thatremains to meet new demands. The “duration” attribute may indicate theperiod of time over which service is to be rendered. The “price(demand)” attribute may refer to a method whose input is the demand by aservice consumer, for a number of service units it requires, whichcomputes the price in virtual currency units, as set by the serviceprovider. For example, the simple service object <<Dell4, CPU, iQ9550,ServerEM015>, 0.1 Ghz, 0.8 Ghz, 2 Ghz, 1 hr, price(x)>, whereprice(x)=1/(2−0.1x)², may be used to describe a CPU service named Dell4,providing an Intel processor of type Q9550 for one hour in units of 0.1Ghz. In this case, a request for 0.5 Ghz (5 units) of this CPU servicewill be priced at price(5)=1/2.25=$0.44 per hour of use.

According to various embodiments, the pricing functions used by simpleservice objects can be flexibly adapted by element managers to reflectdifferent pricing goals and mechanisms. For example, a server may beshared by 10-100 containers, which preferably utilize no more than 50%of its capacity to avoid congestion. In this case, the percentage ofaverage demand to capacity of a given server resource preferably fallsbetween 0.5%-5%.

Consider a commodity service, defined as one where this ratio is verysmall. With supply far exceeding demand, prices will drop to reflectcosts. Thus, a commodity service may be priced at fixed cost-basedprice. For example, suppose the percentage of average demand to capacityfor CPU usage by a container is 0.2%. In such a scenario, the shiftingof a container among servers would have negligible impact on the qualityof CPU services seen by the containers. CPUs can therefore be priced ata fixed level to merely reflect the costs of providing CPUs. In general,a commodity service may be priced at a fixed level, independently ofdemand. However, when the ratio of average demand to capacity issufficiently large, arriving demands may easily deplete the supplyabsent pricing control, thus requiring higher prices to balance thesupply and demand.

A sample pricing function that provides such pricing control is:

price[x]=cost/(1−(U+x)/C)⁴, where C=capacity of the resource; U=amountof resource used; and x=new demand.

Such a pricing function is proportional to costs, penalizing highutilization. When the utilization u=(U+x)/C approaches its limit of one,prices increase rapidly, preventing all but the highest budgetapplications from accessing the resource. For example, supposecontainers require, on average, 2% of the CPU capacity of servers, but20% of their storage I/O capacity. In this scenario, a container wantingto deploy with a server supporting three containers will see thefollowing CPU and storage I/O prices:

price_(CPU)[0.02 C]=cost_(CPU)/(1−0.08C/C)⁴=cost_(CPU)/0.92⁴=1.4×cost_(CPU)

price_(I/O)[0.2 C]=cost_(I/O)/(1−0.8C/C)⁴=cost_(I/O)/0.2⁴=625×cost_(I/O).

Thus, in the above-described scenario, CPU is priced at a relativelysmall multiplier of the cost base of CPU, while the storage I/O ispriced at a relatively large multiplier of the cost base of I/O.Although specific pricing considerations and mechanisms have beendescribed, a large variety of pricing functions may be used according toother embodiments to best reflect specific use considerations.

Composite service objects, which are objects that include more than oneservice object and which relate to the provision of multiple types ofservices, may take the following form according to various embodiments:

<service-identifier, service-1, service-2 . . . , service-n>, whereservice-k is either a simple or composite service object and is referredto as a component of the composite service. In some embodiments, the“duration” attributes of all components of a composite service areidentical, and their common value is called the duration of thecomposite service. For example, a hardware server may be described bythe following composite service object:

<<server-1, Server, LS41>, CPU4, Memory-2, NIC-3, NIC-4, HBA-2> whereMemory-2, NIC-3, NIC-4 and HBA-2 indicate respective simple serviceobjects associated with respective memory services, LAN-interfaceservices provided by two NICs, and SAN I/O services provided by HBA-2.The HBA-2 may itself be described by a simple service object as follows:

<<HBA-2, FC-HBA, Emulex, LP11000-M4>, 0.1 Gbps, 1.1 Gbps, 2.9 Gbps, 1hr, price(x)>.

This service object indicates that the duration of the composite serviceis one hour, as the durations of all components of a composite serviceare identical.

In some embodiments, the price of a composite service is defined as thesum of the prices of all its components. For example, the price of aserver object is the sum of the prices of the units of CPU, memory,network I/O and storage I/O required by a consumer.

The supply chain model databases 246 are maintained by element managers(such as element managers 234, 236, 238, 240, 242, 244 shown in FIG. 2),which handle the service objects corresponding to the respectiveelements that they manage. As explained above with respect to the sampleprocess 300 shown in FIG. 3, according to various embodiments, anelement manager is initialized by the platform manager 250, andsubsequently the element manager proceeds to populate the supply chainmodel databases 246 with respective service objects it is responsiblefor. Once the supply chain model databases 246 have been updated, theelement manager continues to update the dynamic attributes of itsrespective service objects (such as the “used” and “available”attributes). For example, a server manager 238 that is responsible formanaging HBA resources will initialize the supply chain model databases246 with corresponding simple service objects relating to the HBA. Theserver manager 238 will then monitor and update the “used” and“available” attributes of this simple service object by periodicallyaccessing the HBA instrumentation.

As mentioned above, the supply chain economy matches consumers andproviders of resources or services by using pricing and budgeting.According to various embodiments, demand for services is matched tosupply through a shopping model. A consumer element manager (such as oneof element managers 234, 236, 238, 240, 242, 244 shown in FIG. 2),desiring services from a provider element manager, queries the supplychain model databases 246 in search of the best priced provider orproviders of the desired services. The query specifies requirements andthe service or services the element manager is requesting. For example,a query may take the following form:

Query: Server, CPU·units=50 Mhz, Memory·units=4 GB, StorageIO·units=200Mbps, NetworkIO·units=100 Mbps.

Such a query may retrieve records of composite service objects of theservers 214 offering the respective CPU, memory, storage I/O and networkI/O capacity at the lowest price. Once the consumer element manageracquires these records of lowest-priced service objects, it can proceedto extract the identities of the element managers posting these serviceofferings. The consumer element manager may then pursue directinteractions and contract with one or more respective provider elementmanagers to acquire and pay for the desired services. There exists thepossibility that multiple consumers may query the supply chain modeldatabases 246 simultaneously for similar services, and thus potentiallyinterfere with each other's shopping processes. Such interference may beavoided, for example, by providing standard locking mechanisms tomaintain atomicity of the query and purchase transactions.

Moreover, various embodiments may use an auction, or bidding model,rather than a shopping model, to match demand and supply. For example,consumer element managers may post respective bids for services in abidding database, which a provider element manager may then query forthe highest bid price offered for its services and contract to serve it.The shopping model is generally preferred to bidding in situations whereconsumers' demands arrive asynchronously and unpredictably. In suchcases, an arriving consumer can find the low-cost provider by searchingthe supply chain model databases 246. In contrast, a bidding processrequires providers to poll, whether constantly or at intervals, thebidding database to detect arrivals of new bids, while bidding consumersmay be required to wait until enough providers have polled the biddingdatabase and accepted the bids, and thus contract with providers basedat least in part on chance. There are various situations where biddingmay offer benefits over shopping, and those situations may be handledusing the principles described herein.

FIG. 5 is a flow diagram of an example process 500 for deploying a newconsumer element (such as a container) with a provider element (such asa server) in a container system that is used according to variousembodiments for balancing the demand and supply of services. Accordingto various embodiments, the dynamic load balancing approach illustratedby example process 500 provides an effective solution to several of theresource management problems described above. For example, process 500may be used to improve the balancing of demands by containers and thesupply of server resources; it may also be used to balance the resourcebundle allocated to a container, e.g., to match the amount of CPU,memory and storage I/O bandwidth allocated to the container, in order toimprove the use of its virtual budget to best service its resourcedemands.

As shown in FIG. 5, once the relevant consumer element managers andprovider element managers are running, having been initiated by theplatform manager 250, a consumer element manager shops for lowest costprovider for a bundle of services by querying the supply chain modeldatabases 246 as described above (step 502), and contacts the providerelement manager to buy services (step 504). In the case of a containerconsumer, for example, the bundle of services to be purchased mayinclude CPU, memory, and storage I/O.

The provider element manager determines whether the consumer budget issufficient to pay the price for the requested provider services(decision block 506). If it is determined that there is sufficientbudget, the provider element manager deploys the consumer at theprovider, which proceeds to process its workload (step 508). Forexample, CPU and memory resources that have been purchased may beallocated to a container by the underlying scheduler of the containersystem, which may include the use of a traditional operating systemsscheduling algorithm. The server element manager configures thescheduler parameters to accomplish fairly accurate allocation of the CPUand memory. Memory may be allocated by specifying an amount of memory tobe provided. The container system can allocate physical memory, based onthese specifications, or support virtual memory mechanisms that permitover 100% utilization of physical memory. Additionally, the CPU may beallocated by configuring reservations and shares parameters of thescheduler. For example, reservations may be used to allocate a reservedCPU slice, using a time-shared round-robin scheduler, while sharesallocate the remaining CPU bandwidth through a Weighted Fair Queuingscheduler. CPU reservations and shares may be viewed as separateservices, and may be individually priced according to supply and demand.For example, a low-priority application may be unable to buyreservations, and may thus need to settle for shares, which may bepriced lower. A high-priority, mission-critical application, on theother hand, may have sufficient budget to afford sufficient reservationsto support its needs.

Otherwise, if it is determined that there is not sufficient budget, theconsumer element manager initiates a credit check process to decidewhether the consumer can increase its budget or sufficiently lower itsservice demands, and thus continue to run (decision block 510). Forexample, suppose the consumer is a container whose budget is short ofpaying the cost of a provider server. In that case, the container mayuse credit it has accumulated to pay for the service, obtain additionalbudget from the applications it serves, or reduce its demand forservices and the corresponding price to the point where it can afford topay. If one or more of these scenarios is possible, the consumer usescredit, increases its budget and/or lowers its service demands (step512), and the provider element manager thus deploys the consumer at theprovider as described above. Otherwise, if none of these options isavailable, the consumer is suspended and then will either terminate orre-launch when adequate budget becomes available to it (step 514), asdescribed in greater detail below.

After the provider element manager deploys the consumer at the provider,the provider element manager or the consumer element manager monitorsconsumer resource usage and adjusts allocation of resources to optimizeor improve the use of the consumer's budget (step 516). For example, theprovider element manager may find that the consumer is using only 20% ofone service it bought, while using 90% of another service it bought. Inthat case, the provider element manager may reduce the allocation of thefirst service and use the corresponding released budget to increase theallocation of the second resource.

Upon completion or termination of the consumer service period, theprovider element manager notifies the consumer element manager (step518), which may proceed to shop for a new provider offering lowest costservices to meet the consumer's needs (step 520). The consumer elementmanager determines whether the price of the new provider found is lowerthan the price of the old provider (where the consumer resides at thetime), or according to some embodiments, whether it is lower by athreshold amount (decision block 522). Assuming it is, the consumerelement manager moves the consumer to the new provider, in which case itmay also adjust the budget to reflect the price of moving, if any (step524). Namely, according to various embodiments, a price of moving may befactored into the decision making process for whether the consumershould be moved to the new provider, and such price may be subtracted ordeducted from the available budget. Otherwise, if the consumer elementmanager decides to keep the consumer with the old provider, it does notadjust the budget to reflect the price of moving. In either case, theprovider element manager (of the new or old provider) checks to see ifthe consumer budget is sufficient to pay for the provider as describedabove.

According to various embodiments, the process of shopping for a newprovider 520 may depend on specific characteristics of the consumer, theresource, and/or the provider. For example, the containers 120 and 124shown in FIG. 1 may need to exchange high-bandwidth latency-sensitivecommunications through a congested switch in the network 160. Further tothe discussion above, internal I/O pathways (including at either theserver 102 or the server 104) may offer higher bandwidth and lowerlatency, and thus result in improved performance. Therefore, accordingto various embodiments, such internal I/O pathways may be priced lowerthan I/O pathways involving, for example, multiple servers 102 and 104and network 160. The cost of running a workload on a resource provider,in other words, may be adjusted for performance advantages. For example,in one approach, the cost of running a workload on a resource providerbegins with the nominal price charged by the provider, but is adjustedupward if the resource provider delivers performance below a qualitymetric (e.g., average performance) or downward if if the resourceprovider delivers performance above the quality metric. In anotherapproach, the performance metric is considered separately from cost butthe resource budget is adjusted upward if performance is to beconsidered a factor in the procurement decision. In this way,performance can be weighted separately as a decisionmaking factor andthe larger budget will permit selection of a higher-cost butbetter-performing resource provider. As used herein, the term“utilization value” reflects both cost and performance using eitherapproach.

As an example, in the step 520 described above and shown in FIG. 5, theconsumer element manager may determine that it would be more economicalor efficient to move a consumer element from the server 102 to theserver 104 based on reduced I/O pathway pricing. For example, theconsumer element manager may discover that the container 120 should bemoved to the server 104 to obtain one or more resources and communicatewith one or more other elements located at the server 104. This can bethe case where, for example, it is determined at the step 522 that theoverall price of providing container 120 with necessary resources isreduced at least in part because of a lower price of the I/O pathwayshould container 120 be moved to server 104. In that case, at step 524,the container 120 may be moved to server 104 so that the I/O pathwaybecomes more (or entirely) local to server 104, thus benefiting fromhigher expected bandwidth capacity and lower latency.

According to various embodiments, at step 524, the budget of theconsumer element (e.g., container 120) may also be adjusted (e.g.,increased or decreased) based at least in part in such change inpricing. As indicated above, in an alternative embodiment, the pricingof resources (e.g., associated with the I/O pathway) may be increased toaccount for performance improvement that would result from movement of aconsumer element to another server and the resulting localization.

According to other embodiments, the process of shopping for a newprovider 520 may depend on functional characteristics of the consumer orprovider. For example, the server 102 may be used to support developmentof containerized applications. The server 104—the provider, forexample—may be used for testing the containerized application 130—theconsumer, in this example. The process 500 may be used to select a newprovider (the server 104), from among a group of servers providing restsof containerized applications, to run tests (consumer) of thecontainerized application 130. Similarly, the server 104 may be aproduction system running containerized applications and the process 500may be used to dispatch the containerized application 130, and itscontainer 120, from the development server 102 to the production server104.

The order of steps in the example process 500 described above isillustrative only, and the steps can be carried out in a differentorder. Moreover, it is contemplated that modifications and extensions ofthe process 500 will be used according to various embodiments. Forexample, a consumer may need to contract with two or more providers tobe deployed, as in the case of a container that needs to acquire abundle of resources offered by a server as well as SAN switch bandwidthand storage space at a storage array. In such scenarios, deployment ofthe consumer can be supported by extending step 502 to shop for multipleproviders and then repeating the remaining steps for each of theseproviders. Additionally, for example, as explained below with respect toFIG. 6, the example process 500 shown in FIG. 5 may be modified orextended to enable the adjustment of resource allocations to obtaindesired SLAs.

According to various embodiments, the above-described supply chaineconomic principles may also be used to manage software licenses, suchas temporary (time-limited) software licenses. For example, regardlessof type (such as authorizations of software use per user, per CPU, perserver, or per container), licenses may be modeled as resources to bepurchased by an application manager 234, much like other resources thatit may purchase from the container 212. License element managers (whilenot shown, may be included as part of the platform layer 230) may beused to set the prices of the licenses based on costs and demands. Inthis manner, license management may be greatly simplified and unifiedwith the allocation of other types of resources. For example, anapplication that is unable to acquire a needed license may suspend itsoperations and release its resources, as explained below, thusincreasing the overall efficiency of the system. Additionally, licensesmay be more efficiently used, since in situations where the licenses arehighly utilized, they will be allocated to high priority tasks, whilelower priority tasks may be suspended until they can afford thelicenses. As soon as a license is no longer needed, it may be releasedand available for other tasks. Additionally, an administrator mayconsider the ROI of licenses, as with other resources, to plan theexpansion, or contraction, of licenses capacity. For example, if alicense's ROI is above a certain threshold, it may be desirable toacquire more licenses to increase the supply to meet demand.

FIG. 6 shows an example process 600 for delivering service-levelagreement targets through resource allocation in a container system,which includes many of the steps of process 500 shown in FIG. 5 anddiscussed above. Although not required, for the purpose of simplifyingthe following description, it is assumed that the target service-levelagreement relates to an application running on a container. However, theservice level of other types of computer elements may be controlled inthe following manner according to various embodiments.

Following the initial monitoring of resource utilization and optimizingof the container's budget (step 516), it is determined whether theconsumer service period has terminated (decision block 602), in whichcase the provider element manager notifies the container element manager(step 518) as described above. Otherwise, the container element managermonitors and obtains the value of the SLA parameter of interest, such asthe average transaction rate of an application, the average transactiondelay of an application, the average communications latency of theapplication, or the number of transactions performed within apredetermined prior time period by an application (step 604). Forexample, an application element manager may monitor the value of the SLAparameter, through respective instrumentation, and inform the containerelement manager of the SLA parameter. The application may define its SLAgoal as 100 transactions per second, in which case the SLA parameter ofinterest is transaction-rate. In general, because SLA parameters can beassumed to increase monotonically with the amount of resources allocatedto an application, the management of SLAs may be accomplished asdescribed herein by finding a budget and a respective resourceallocation that will accomplish the target SLA value.

The container element manager determines whether the SLA parameter ofinterest is below a desired target (decision block 606), in which case,for example, the application's payments to the container (e.g., ofvirtual currency units) are increased such that the container's budgetis increased, and it is able to purchase more resources to increase theSLA parameter of the application (step 608). After such an increase, thecontainer's budget use is again monitored and optimized or improved asdescribed above.

If the container manager determines that the SLA parameter is at orabove the desired target, it is determined whether the SLA parameterexceeds the desired target by more than an acceptable threshold(decision block 610), in which case the payments are reduced, thusreducing the container's budget and the resources it buys, saving onapplications costs, and keeping the SLA performance within a desiredtolerance range (step 612). After such a reduction, the container'sbudget use is again monitored and optimized or improved as describedabove. If the SLA parameter is within the acceptable range, however, areduction is not applied, and the process is repeated until it isdetermined that the consumer service period has been completed orterminated.

According to various embodiments, the process 600 for deliveringservice-level agreement targets through resource allocation in acontainer system may be modified, adapted, and/or simplified for certainresources and SLA metrics. For example, in the case of allocation of I/Opathways to reduce or minimize latency, the process 600 may be modifiedas follows. The SLA parameter may be selected as the latency-hop-count,e.g., the number of physical switches traversed by an I/O pathway. Forexample, I/O pathways between elements located, or resident, at the sameserver (e.g., the containers 120 and 122 in FIG. 1) generally do nottraverse any physical switch, and thus may be described as having alatency-hop-count of 0. Such I/O pathways may also be referred to ashaving Class-0 Latency SLA. On the other hand, I/O pathways betweenelements located or resident at different servers (e.g., the containers120 and 124 in FIG. 1) and attached to a common switch (e.g., a commonswitch of the network 160) may be described as having alatency-hop-count of 1, and may be referred to as having Class-1 LatencySLA. According to various embodiments, an I/O pathway may involve two ormore physical switches, and may be described as having alatency-hop-count of 2 (or more) and referred to, for example, as havingClass-2 Latency SLA.

According to various embodiments, the latency-hop-count associated SLAvalue may be described with respect to the ordinal preference {Class-0,Class-1, Class-2, . . . Class-n}, where Class-0 is preferred to Class-1,Class-1 is preferred to Class-2, and so on to the extent additionalClasses are defined. With respect to the process 600, a comparison canbe made between a Target Latency Class and an Actual Latency Class(e.g., Target=Class-0, Actual=Class-1) at step 606. If the ActualLatency Class does not meet the Target Latency Class, payments to theconsumer (e.g., the container) may be increased at step 608, and,following return to step 516, an I/O pathway can be acquired that candeliver the Target SLA Value (e.g., Class-0). For example, the process600 described with respect to FIG. 6 can be modified in a mannerconsistent with the above description so as to simplify the monitoringand control of SLA values to classification of the I/O pathway intolatency class.

It will be understood that the SLA-delivery process 600 described abovemay be flexibly adapted to achieve various goals, such as improving itshandling of stochastic fluctuations of an SLA parameter. For example,the steps of increasing (step 608) and decreasing (step 612) payments bythe application to the container may use standard mechanisms ofStochastic Approximation theory, including the Robbins-Monro orKiefer-Wolfowitz algorithms, to regulate the changes in payments toassure convergence. Such a design may be implemented, for example, toachieve more desirable results in connection with non-monotonic SLAparameters. For example, an embodiment using a Robbins-Monro proceduremay replace steps 606-612 with the following iteration:

R(n+1)←R(n)+a(n)[SLATarget−SLAParameter(R(n))]

where n is a counter of the iterations, R(n) is a vector describing theresource bundle allocated after n iterations, SLATarget is the desiredvalue of the SLAParameter, and SLAParameter(R(n)) is the observed valueof the SLAParameter after n iterations. The vector a(n) represents theincrease/decrease of resources through the n-th step of the iteration;typically a(n)=a/n, where a is a fixed bundle.

Although the SLA-delivery process 600 described above uses an economicmodel and virtual currency units to control SLA levels, other manners ofcontrolling SLA levels may be used according to various embodiments. Forexample, the allocation of resources to a container, or to anapplication, may be independent of any economic budget or transfer ofvirtual currency units, and may instead be based on other measures of anapplication's or container's importance.

The process 500 described above may also be modified or extendedaccording to various other embodiments. For example, since currentcontainer systems are not readily adaptable to handling the managementof storage I/O through HBA or storage systems schedulers, as analternative to an arbitrary first-come-first-serve process, the process500 described above may be modified or extended as shown in FIG. 7 tofacilitate the handling of storage I/O.

FIG. 7 is a flow diagram of an example process 700 for economicallybased I/O scheduling in a container system, which includes many of thesteps of the process 500 shown in FIG. 5 and discussed above. Althoughnot required, for the purpose of simplifying the following description,it is assumed that the consumer is a container, the provider is aserver, and the resource is storage I/O. It will be understood that,according to alternative embodiments, the resource being managed may beother types of I/O, such as network I/O.

Following the deployment of the container at a server (step 508), theserver element manager monitors storage or network I/O usage by one ormore containers, such as by collecting data from one or more of thecontainer system, the HBAs (step 702), or the NIC. According to variousembodiments, the server element manager may be configured to preventcongestion along storage I/O pathways, as might occur in cases of usagelevels approaching the capacity limits. For example, the server elementmanager may prevent congestion by using pricing functions as describedbelow that increase prices dramatically when utilization approaches 50%of the capacity.

The server element manager optimizes or improves the resources allocatedto containers, as described above (step 516), such that containersacquire a share of the storage I/O resources that is commensurate withand optimally reflects their budget. The server element manager thenperiodically estimates both the average storage I/O capacity used andthe average available I/O capacity, and updates the respectiveattributes of the storage I/O objects in the above-described supplychain model databases 246 with this usage data (step 704). It is notedthat the usage data reported to the supply chain model databases 246will impact price computations, with excessive utilization of storageI/O capacity resulting in respective price increases, and higher pricesin turn deflecting demand by new or existing containers to servers withlower utilization (and prices) of storage I/O. For example, pricecompetition over using storage I/O resources may result in migration oflow budget containers from overloaded servers to other servers wherestorage I/O resources are more highly available, and are thus pricedlower. Higher priority containers, on the other hand, may use theirhigher budgets or credit to obtain a preferential share of storage I/Oresources.

The server element manager also computes the actual (versus projected)costs expended by each container, and applies these prices to handle itscurrent commitments to containers (step 706). For example, higher usageof storage I/O results in higher prices and immediate costs assigned tocontainers, such that containers of lower priority and high storage userequirements may quickly exhaust their budget or credit and be suspendedor terminated, as described below. In this manner, the low prioritycontainers relinquish storage I/O capacity to containers having a higherpriority and, thus, a higher budget.

Based on the computed costs, the server element manager evaluateswhether the container's budget is sufficient to pay the cost (decisionblock 708). If it is, the service period of the container continuesuntil it ends, and the server element manager notifies the containerelement manager of the completion of the service period (step 518).

Otherwise, if the container's budget is not sufficient, the serverelement manager evaluates whether the container's credit (costs minusbudget) exceeds an acceptable credit threshold (decision block 710).According to various embodiments, high-priority containers may havehigher budgets and credits and can thus afford to overpay the serverelement manager to guarantee that they do not run out of storage I/Oresources. If it is determined that the container's credit exceeds thethreshold, the container element manager initiates a credit checkprocess to decide whether the container can increase its budget orsufficiently lower its service demands, and thus continue to run(decision block 712). If possible, the container makes any necessaryadjustments (such as a budget increase in the case of high prioritycontainers, or reduced service demands) and continues to run (step 714),until the service period has ended and the server element manager hasnotified the container manager of the termination of the service periodas described above. Otherwise, the server element manager suspends orterminates the container execution and notifies the container elementmanager, which becomes responsible for addressing the suspension ortermination (step 716).

Upon termination of the service period and notification to the containerelement manager, the server element manager reports usage data to thecontainer element manager and settles any credit, overpayments orunderpayments with the container element manager (step 718). Thecontainer element manager may then proceed to shop for a new serveroffering lowest cost services to meet the container's needs (step 520),as explained above.

The economically based scheduling process 700 described above may beused effectively to de-correlate peaks of competing, bursty I/O flows.For example, consider the scenario of four containers sharing a commonserver and a 4 Mbps Fiber Channel HBA, where the containers generateaverage storage I/O flows of 250 Mbps, 250 Mbps, 200 Mbps and 300 Mbps,respectively. The aggregate demand average of 1 Gbps consumes only 25%of the HBA capacity. A resource scheduler may limit its consideration toonly the average demand which, in this case, would be manageable by theHBA and SAN. However, consider an alternate scenario where the I/Otraffic streams are bursty, with a peak/average ratio of five for eachcontainer. If the four I/O streams associated with the containers areuncorrelated, their peaks will be likely dispersed and the peak of theaggregate stream will generally be less than 2 Gbps, which can behandled by the HBA and SAN with negligible or relatively few queuingdelays. However, if the I/O streams are correlated, their peaks may becompounded to generate, for example, up to 5 Gbps peaks, utilizing 125%of the capacity and generating sustainable congestion, delays, andlosses. The scheduling process 700 described above reduces thelikelihood of compounded peaks, since they result in peak prices and acorresponding depletion of budgets and credits of low budget containers,leading to suspension, termination, or migration of such containers toservers with lower storage I/O prices until they find servers wheretheir peaks are sufficiently de-correlated from other containers.

Thus, the allocation of containers to common servers according to thescheduling process 700 may result in substantially de-correlated peaksand substantially reduce the peak/average ratio seen by servers. Forexample, consider the example of four containers above. If their peaksare uncorrelated, the peaks of the aggregate stream will generallyrequire at most 1.5 Gbps (the peak of the largest component stream),while their average traffic is 1 Gbps. The burstiness ratio(peak/average) of the aggregate stream 1.5/1=1.5 therefore representsonly 30% of the burstiness of the individual streams (1.5 divided by 5).The economically based scheduling process 700 described abovesubstantially reduces interference not only between traffic averages,but it also reduces the interference between correlated traffic peaks.This results in smoother, less bursty, aggregate workloads, which maypermit more efficient processing.

It will be understood that, according to various embodiments, theprocess 700 described above to manage storage I/O flows may be appliedto other forms of I/O, such as network I/O. For example, the abovedescription should be understood to include alternative processeswhereby references to “storage” are replaced by references to “network.”It will similarly be understood that storage I/O flows typically utilizenetwork-PO flows, such as Ethernet (e.g., Fibre Channel over Ethernet(FCoE)), Transmission Control Protocol/Internet Protocol (TCP/IP) (e.g.,Network File System (NFS)), and SAN (e.g., Fibre Channel (FC), InternetSmall Computer System Interface (iSCSI)), to transfer information suchas storage access commands. The scheduling process 700 is thereforeindependent of the specific underlying network, and of the specificaccess commands carried by the described flows. Accordingly, the process700 may be applied to schedule network I/O flows and thereby providesimilar or identical benefits to those associated with storage I/Oflows, such as smoothing the peaks of bursty traffic and/or supportingpriority services.

The order of steps described above with respect to scheduling process700 is illustrative only, and can be done in different orders. Moreover,the aforementioned beneficial effects are true not only for I/O streams,but for workloads sharing other resources as well.

The contracting of services between a consumer and a provider, asdescribed in the example processes above, may include the use of astandard request-response protocol (such as SOAP) to submit a purchaseorder to the provider and transfer a respective payment. In response,the provider may deploy the service requested by the consumer andrespond with a service confirmation.

FIG. 8A is an example purchase order data structure 800 issued by aconsumer element manager for use in purchasing services from a providerelement manager. The first two fields of the data structure 800,source-ID field 802 and provider-ID field 804, respectively identify thesource consumer and destination provider. The third field,transaction-ID field 806, identifies the particular purchase order. Thefourth field of the data structure 800, service field 808, identifiesthe service and provides parameters to quantify the purchase. The fifthfield of the data structure 800, payment field 810, provides paymentdata including payment amount and authentication data to establish thevalidity of the payment. Finally, the sixth field of the data structure800, authentication field 812, provides data to authenticate thevalidity of the purchase order transaction.

FIG. 8B is an example service confirmation data structure 850 issued bythe provider element manager for use in confirming or rejecting thepurchase of services by the consumer element manager. The first threefields of the data structure 850, source-ID field 852, provider-ID field854 and transaction-ID field 856, correspond to the first three fieldsof the data structure 800 described above. The fourth field of the datastructure 850, service confirmation field 858, includes data to confirmthe service and enable the source to access it. Alternatively, assumingthe provider has rejected the transaction, the service confirmationfield 858 would include data with the reason or reasons for rejection,such as insufficient resources or a price change. Finally, the fifthfield of the data structure 850, authentication field 860, provides datato authenticate the validity of the service confirmation.

As described below, various embodiments may also be used to address theproblems of container sprawling and energy consumption in containersystems using supply chain economics. Regarding sprawling, as explainedin greater detail below, these embodiments may be used to suspend orterminate containers that are no longer needed or productive. Theseembodiments may also be used to terminate containers, or to disallowtheir re-activation if in a standby state, that are determined to beinconsistent with the current versions of their container system andapplications. Regarding energy consumption, these embodiments may beused to consolidate and shift containers into fewer servers, forexample, while still providing desired SLA performance, and switchingother unused or non-productive servers OFF or into standby mode toreduce energy use. The supply chain software model and processesdescribed above provide mechanisms and metrics to quantify howproductive or non-productive a service element is.

The following description details an example process 900, shown in FIG.9, for managing the states of container system elements, which asexplained further below, may be used to address sprawling and energyconsumption issues. For simplicity, the following description assumesthat the system element is a container, although the general principlesthat follow may be readily adapted for any type of system element.

A container is first initialized, for example, through the use of aninitialize signal generated by a management station (step 902) or anautomated action of a container manager. Similarly, for example, anapplication element may interpret events generated by a launch as aninitialize signal. After being initialized, the container attempts toobtain an initial budget to acquire resources for its operations (step904). It is next determined whether the container was successful inobtaining an initial budget (decision block 906), in which case thecontainer tries to acquire the resources needed to launch a respectiveservice component (step 908). Otherwise, it begins the terminationprocedure by releasing any resources allocated to it (step 910). If thecontainer is successful at acquiring resources (decision block 912), itis provisioned, deployed, and remains in an active state (step 914)until it receives a signal to switch the service element OFF to an idleor standby state (step 916). After the terminate signal has beenreceived, the container begins the termination procedure by releasingresources allocated to it, as described above.

On the other hand, if the container is not successful at acquiringresources, the container will wait an amount of time for sufficientresources to become available before attempting to acquire resourcesagain (step 918). For example, during this waiting period, the containermay use an exponential “backoff” mechanism, whereby the containerrepeats its attempts to acquire resources, but doubles the waitingperiod between repetitions with every failure. If it is determined thatthe container should continue to try to acquire resources (decisionblock 920), it will do so as described above in step 908. Otherwise, forexample, if failures persist beyond some timeout period, the containerabandons attempts to launch and begins to terminate.

Once resources have been released, it is determined whether thecontainer should remain in a standby state (decision block 922), inwhich case the execution of the container stops, but it remains in asuspended or standby state and retains sufficient state data, forexample, by using storage services to retain state data in image form,and for which the container may be required to pay (step 924).Otherwise, the container terminates execution and may be deleted (step926).

According to various embodiments, the applications being executed by thecontainer are first terminated, and then the container is terminated.Such a graceful termination may be pursued through a recursivetermination of the supply chain elements supported by the container. Forexample, a container element manager may issue a terminate signal to acorresponding operating system manager, which propagates the signal toan application manager, which in turn signals termination to isapplication. The application may then begin the termination steps asdescribed above with respect to the process 900, after which atermination complete signal to the application manager, and is forwardedto the operating system manager, which in turn sends a terminate signaland receives a termination complete signal back from the operatingsystem. Finally, the operating system's termination complete signal maybe forwarded to the container manage, which can signal the container toterminate. It will be understood that terminating (or even suspending) acontainer operation may result in damages if conducted improperly or atan inappropriate time. Thus, according to various embodiments, anotification procedure may be invoked to notify administrators ofpending terminations or suspensions, such that termination or suspensionmay only be completed once administrator permission has been received.

For a container in standby state, it is determined whether terminationshould follow (such as by receipt of a terminate signal) (decision block928), in which case the container terminates execution as describedabove. Otherwise, for example, if it is determined that the containershould re-activate, the container seeks to obtain a budget to acquireresources for its operations as described above, for example, uponreceiving an initialize signal. It will be understood that the specificactions described above in connection with process 900 may be modifiedfor non-container system elements, and that the order of steps inprocess 900 are also illustrative only.

According to various embodiments, a process such as process 900described above may be used to control container sprawling by suspendingor terminating non-productive system elements, such as containers. Forexample, consider the ROI of a container, which measures therelationship between the payments it collects from applications and theprices it pays for underlying server and I/O resources. If thecontainer's ROI is greater than one, the container is earning more thanit expends, and the container may be classified as being productive increating applications value that exceeds the costs of theinfrastructures it uses. However, if the container's ROI is less thanone, this means that the container produces less value than the cost ofresources it consumes, and the container may thus be classified asnon-productive. In this manner, ROI is one example of a metric ofproductivity that may be used in determining whether a system elementshould be suspended or terminated, or whether it should remain active.

A process such as process 900 described above may be used to assure, forexample, that applications' budgets are sufficient to keep one or morecontainers' ROI greater than one, and to notify applications'administrators (element managers) as needed when budgets are low. It theROI of one or more containers remains less than one for more than athreshold period, for example, it may indicate that an application'sbudget is too low to sustain productive operation, and that thecorresponding, non-productive container should be suspended orterminated. For example, a container may receive a terminate signal toswitch it OFF to an idle or standby state (per step 916 of process 900described above) as soon as the container's productivity level or score(for example, measured by its ROI) has been determined to be less thanone for a predetermined time period. Additionally, for example, thelength of time that the container's ROI has been less than one may be afactor in deciding whether the container should be terminated, or onlysuspended for the time being.

As with the sprawling issue, the process 900 described above and similarprocesses may also be used for energy management. For example, suchprocesses may be used to suspend or terminate (switch OFF) servers thatare classified as being non-productive, as in the case where a server'sROI is less than one for a sufficiently long period of time. In thiscase, the server element manager, much like the case of the containermanager described above, can monitor the ROI and detect termination orsuspension conditions. The server manager may then pursue a terminationprocess, similar to the recursive termination process described above,where all containers on the server are first terminated, or moved toanother server, before the server manager suspends the server intoStandby state (so as to consume less energy and cooling resources, forexample) or switches the server OFF.

According to various embodiments, process 900 and similar processes mayalso be used to assure consistency of a suspended container with changesin applications. For example, the container manager may prevent suchinconsistencies by sending a terminate signal, as described above, toall containers whenever their respective operating system orapplications software has changed, thus causing the applicablecontainers to transition from standby to terminate state, at which pointit may be deleted.

Although the above descriptions consider a single-domain containerenvironment, it will be understood that the principles described hereinmay also be applied to multi-domain environments, e.g., a multi-cloudenvironment. For example, FIG. 10 is a block diagram of an examplemulti-domain software system environment 1000 for managing virtualizedresources in “multi-cloud” systems. According to various embodiments, asshown in FIG. 10, container environment 1000 includes two examplesoftware systems 1002 and 1004, each of which is similar to the moredetailed example software system 200 shown in FIG. 2, and which operatein a first and second domain, respectively.

As shown, the software system 1002 operating in the first domainincludes a user interface subsystem 1006 and one or more functionalmanagers 1008 and 1010. Together, these elements make up a functionalmanagement layer 1012 of software system 1002, and provide specificmanagement applications as described above in connection with FIG. 2.Software system 1002 also includes one or more element managers 1014 and1016, which monitor and control one or more respective container stackelements 1018 and 1020. The software system 1002 also includes one ormore databases 1022 (such as the supply chain databases 246 andoperations databases 248 described with reference to FIG. 2), as well asa platform manager 1024. These elements are included in a platform layer1026 of the software system 1002 to provide the infrastructures formonitoring the container stack elements 1018 and 1020, modeling thesecontainer stack elements as part of a supply chain economy, andcontrolling the operations of the container stack elements, as describedabove.

The software system 1004 operates in the second domain, includes similarelements as the software system 1002, and also includes a proxy manager1030. According to various embodiments, the domain software system 1004exports one or more resources or services to the domain software system1002 by using the proxy manager 1030. The proxy manager 1030 exportsinstrumentation to monitor and control these provided resources to oneor more of the element managers 1014 and 1016, such as container elementmanagers, of the first domain software system 1002. The first domainsoftware system 1002 may view the second domain software system 1004 asa service element integral with its supply chain model.

According to various embodiments, the second domain software system 1004is in complete control of the resources (or services) and capabilitiesexported to the first domain software system 1002. For example, thesoftware system 1004 may be an external cloud provider exporting rawserver services to the software system 1002. In this case, the softwaresystem 1002 can access these services, using its local element managers1014 and 1016, to allocate, for example, CPU, memory, and storageresources at the second domain software system 1004 and then monitor andcontrol their use and operations.

Moreover, according to various embodiments, software systems 1002 and1004 are separately owned and/or managed. For example, software system1002 may be owned and operated by a small business that experiencessteady computing needs except for two hours in each day, during whichtime its computing needs are consistently elevated. In this case, ratherthan purchasing permanent computing resources to handle the two hours ofelevated needs per day, for example, software system 1002 may lease orpurchase additional computing resources from software system 1004 (e.g.,owned by Amazon.com, Inc.) on an as-needed basis and transfer excessworkloads to software system 1004 (“bursting”). For example, computingresources from software system 1004 may be leased or purchased tofacilitate the execution of a multi-tier web service by a cluster ofcontainers (or applications). In that example, the software system 1002may lease or sell resources from software system 1004 to execute thiscluster of containers (or applications) and then migrate the containercluster (or application cluster). For example, the migration may takeplace from a private cloud of a small business to the public cloud ofanother business (e.g., of Amazon, Inc.). It is noted that, according tovarious embodiments, even if needed computing resources are availablefrom within software system 1002, such resources may be purchased fromsoftware system 1004 based on relative price offerings.

The asymmetric relationship between software systems 1002 and 1004 shownin FIG. 10 and described above may be extended to provide full symmetry.In that case, the first domain software system 1002 would incorporateits own proxy manager (not shown) to export services to the seconddomain software system 1004, which would integrate it within its supplychain through one or more of its respective element managers.

The supply-chain principles discussed herein may be used to scalecontainers up/down, by adding or removing resources to existingcontainer components, or to scale containers out, by adding morecontainer components or suspending containers. The decisions to scaleup/down and out are based on a supply chain 1100 outlined in FIG. 11,and the revenues and expenses of each of the entities involved in thatFigure. The supply chain 1100 may also be used to determine the sizingand placement of containers, when a selected container is deployed, andto determine future sizing and placement requirements based onanticipated changes in container load.

As indicated above, the systems, apparatus and methods described hereincan be applied, for example, to recommend and eventually migrateworkloads back and forth among multiple providers (private and/orpublic) in a cloud environment. This includes provisioning a private orpublic data center, migration from on-premises data centers or otherprivate cloud providers to public data centers or cloud providers (alsoknown as public cloud service providers or platform, or public CSPs),migrating back, and so on. This includes, for example, cloud bursting,whereby a workload running in a private cloud or data center “bursts”into a public cloud when there is demand or need for additionalcomputing resources. This also includes scenarios where only publiccloud providers are used (e.g., no access to private data centers).

The principles of the present invention also apply to migration amongdifferent public data centers or cloud providers. For example, anapplication or other workload may be migrated between and among privateand/or public cloud providers based on the price of resources beingcharged by each. According to various embodiments, resources areconsidered a commodity, and the pricing or cost available from multipleproviders is considered as the primary or sole factor in decidingwhether to migrate.

For example, the systems, apparatus and methods hereof may incorporate acommodity referred to as “Cost” which is sold in a private and/or publicvirtualization or cloud marketplace. According to various embodiments,with respect to providers in the cloud, the value of the Cost employedis the monetary (e.g., dollar) cost charged by the cloud provider to runthe current load on it; the capacity of the Cost is the monetary (e.g.,dollar) budget specified by a user for running the load in the publiccloud; and/or other commodities have extremely large, unlimited orinfinite capacity, making the Cost the dominant commodity in the price.Particularly with respect to providers in a private data center, thevalue of the Cost may be the monetary (e.g., dollar) value of runningthe load in-house; the capacity of the Cost is extremely large,unlimited or infinite, making the rest of the commodities dominant;and/or in a non-hybrid environment, the system can operate in theabsence of available providers in the cloud. With respect to providersin both private data centers and the public cloud, the Cost can includethe cost of moving (both from internal deployment to the cloud, and fromone cloud provider to another); the pricing function (e.g., 1 divided bythe square of (1−utilization)) can be set such that the closer an entityis to the budget allocated for running on the cloud, the more expensiveit is to move to that cloud.

Although sales and transactions according to the principles discussedherein are contemplated in various embodiments to be tied to realcurrency (e.g., U.S. dollars), it will be understood that any suitabledenomination or unit of currency, including virtual currency, physicalcurrency, or electronic currency (e.g., Bitcoins), whether or not tiedto any government-issued or “real world” monetary unit or system, may beused. It is also contemplated that one form of currency may be laterconverted to another form of currency (e.g., U.S. Dollars to CanadianDollars, or U.S. Dollars to virtual currency units, or virtual currencyunits to U.S. Dollars, to name a few examples).

According to various other embodiments, price or cost is one of multiplepotential considerations in choosing a cloud provider, any one or moreof which can be relied upon or weighed. For example, the additionalconsiderations in choosing one or more cloud providers includereputation, past performance, actual or anticipated environmentalimpact, the existence of preferred vendors or providers, utilization(including real-time utilization), contractual clauses or restrictions,quality of service (QoS) metrics, requirements or guarantees, compliancerequirements, regulatory requirements, pricing discounts, securityconsiderations, and/or performance or other metrics. One or more ofthese considerations may be additional or secondary to price or cost, ormay supplant price or cost as the primary and/or sole consideration(s).Based on one or more of the foregoing, the principles discussed hereinallow the creation of “price performance” metrics that can be used toreview and choose among cloud providers.

According to various embodiments, real-time migration, placement and/orconfiguration of an application or other resource consumer isaccomplished through decision-making that is automatic or semi-automaticof manual intervention. For example, an entity in the cloud market caninitially purchase resources from a public cloud based on the real-timeprice (or spot-price) offered by competing cloud providers. Such initialpurchase may be followed by a medium or long-term contract for one ormore resources, including those initially purchased and assigned.

It will be understood that the principles discussed herein apply notonly to initial placement of applications or workloads with one or moreproviders, but also a recurring, periodic or continuous monitoring ofavailable providers. For example, once a certain demand has beenaccounted for through deployment or migration to a cloud provider, theprinciples disclosed herein can be employed to continuously exploreand/or shop for alternative providers that may provide one or morebenefits over the initially selected provider. This may include bringingan application or workload back to an on-premises or private provider.

In addition to (or in lieu of) movement between cloud providers, theprinciples disclosed herein can also be applied to recommend ordetermine when an application or workload should be resized or cloned,whether on the same or different cloud, for example, to optimizeperformance or operating cost. These principles can be used not only tomanage cost and/or performance, but also to reduce the risk ofunintended or unanticipated charges.

According to various embodiments, these principles of migrationmanagement are controlled through a single visual or computinginterface. This interface may include budgeting controls, performancecontrols, compliance controls, and the like. It may also provide asummary or visualization of available cloud providers and metricsassociated with one or more such providers. For example, a userinterface (UI) may be provided which provides visual or graphicalrepresentations of public and/or private cloud providers. According tovarious embodiments, the UI will display one or more of the following:target providers supported by a customer; a list of one or more targetsthat the customer belongs to; the discount available from one or moreproviders; the customer's budget; the total mount currently spent by thecustomer on providers; or a list of users in the customer's account.

Moreover, the principles discussed herein can be used to managetrade-offs between quality of service (QoS), performance and cost. Whenperformance is the focus, for example, the systems, apparatus andmethods can be used to move an application or workload to the best orhighest-quality cloud provider, or balance performance with cost.

The principles described herein can be used to facilitate initial clouddeployment, enabling efficient scale-out of workloads, as well asoptimal distribution considering data locality and its impact onperformance. They also have the benefit of encouraging pricing andperformance competition among providers, including public cloudproviders of resources. According to various embodiments, pricing isused in addition to (or in lieu of) performance metrics to differentiateamong two or more cloud service providers (CSPs). Thus, decisions can bebased on a tradeoff between price offered by a CSP and its historicaland/or anticipated performance metrics. For example, when decidingbetween deployment with competing first and second CSPs, the systems,apparatus and methods can be used to weigh the benefits and tradeoffsbetween selecting a first CSP having superior performance andcommensurately higher price of a resource (e.g., CPU) versus a secondCSP having inferior performance and a lower price for the resource. Itwill also be understood that principles associated withcontainerization, as discussed herein, for example, may be employed tomigrate and move applications or workloads to the cloud.

Cloud-based resources can be procured by a consumer in baskets orbundles, which may correspond to the minimum needs of a consumer (e.g.,CPU and memory needs) or may instead correspond to standard packageofferings available from one or more CSPs or other sellers. (Asdiscussed in greater detail below, a seller may be an outside vendor ormay instead be an internal resource provider.) For purposes both ofactual procurement and computing ROI to make a procurement decision,these “templates” may, in some embodiments, be treated as the mostgranular procurement unit—that is, a seller sells templates and aconsumer buys one or more templates, typically from a single seller.When making procurement decisions as described above, a consumer maycompare offerings of a particular template among CSPs, but may alsocompare template pricing to the on-demand price of the templateconstituents. If the individual constituents are priced less expensivelythan the template, the consumer may move from the template procurementmodel to procurement of individual components. If the resources providedin a template exceed the consumer's actual needs, the comparison may bebetween the cost of the template and the cost of the essential resourcesif procured on demand. Templates typically bundle commodities such asmemory, CPUs, databases, network bandwidth, and I/O resources. Memorymay be expressed in terms of tiers, with each tier corresponding todifferent performance levels (e.g., response time). A template iscompetitive for a consumer's needs if all template resources meet orexceed the minimum requirements of the consumer (and do not exceed amaximum requirement, if any).

Additional benefits are also achievable using the principles discussedherein. For example, the more utilized a private data center is, themore lucrative the public providers may become; the spot-pricefluctuations may directly or indirectly affect market decisions (e.g.,the more expensive a spot-price is, the more expensive price theproviders can opt to quote); if there is no congestion in a local datacenter, the costs associated with moving may inhibit or prevent demandfrom moving to the public cloud; once a data center becomes congestedand the costs associated with moving is less than the difference inprice, it may become cheaper to move to the public cloud; once thebudget allocated for running on the cloud is close to being met (e.g.,because there is sufficient demand running consistently on the publiccloud providers), the prices of running on them would become highercausing demand to remain with private or on-premises data centers (and,for example, the market can recommend provisioning new hosts). Thus, forexample, placement decisions may consider the overall congestion of theprivate data center or data centers, the current spot-price associatedwith one or more public cloud providers, and/or the cost of migration,among other factors.

These and other benefits, which will be apparent to persons of skill inthe art, can be used to improve perform and/or system efficiency, andalso overcome potential challenges, as public clouds continue to grow inavailability and demand, and as they continue to handle more workload inthe market. The principles described herein provide additional benefits,particularly given that cloud-provider pricing is often complex andconfusing; there may be a large number of choices an entity must make ortake into account before creating instances or making migrationdecisions; performance may be unclear and not guaranteed; and costs(e.g., bills) may come as a surprise. The principles described above canreduce uncertainty and avoid or limit surprises, and the analysis canconsider the on-demand price for the needed resources, the cost whendeployed with a CSP, and the effect of personalized (e.g., long term)customer contract pricing.

Cloud-based resources can also be the subject of discounts, coupons orreserved instances, i.e., promotions that offer a basket of servicesthat may be discounted and/or subject to constraints, such as temporalconstraints, or other restrictions as described below. Once a reservedinstance expires, the resources are no longer available on a discountedbasis; but prior to that time, the resources are available at thediscount either currently or in the future (in the manner of an option).In identifying procurement needs, a consumer will take reservedinstances into account in selecting a provider/seller. However, areserved instance often can be transferred to another consumer, whichprovides greater flexibility. In such cases, the consumer holding areserved instance can behave as buyer and seller, comparing availabletemplates and/or on-demand offerings against the reserved instance andoffering it for sale to other consumers. In some instances, the lowestoverall cost for multiple consumers will occur when a reserved instanceis shifted to another consumer and the resource deficit is filled with atemplate and/or on-demand services.

In some cases, reserved instances may not match the workload, in whichcase the associated discount may be reduced commensurately. For example,if an available reserved instance exceeds the requirements of a VMworkload, it may be possible to use the reserved instance only to theextent of the needed resources with a reduction in the associateddiscount; this may still result in more advantageous pricing than theon-demand alternative. Alternatively, the unused resources may beallocated to a different buyer if the reserved instance is notrestricted to a single buyer, in which case the discount is fullyapplied to the reserved instance. Conversely, if the reserved instanceis insufficient to fully service the workload, the overall discount ispartial—that is, the reserved instance cost reflects its full discount,but since the workload requires additional, undiscounted resources, theoverall discount (from the point of view of the workload) is partial. Inthis regard, it should be understood that while it is often possible tosplit up a workload in this way, so that a discrete portion may run onand fully utilize a reserved instance (with the remainder of theworkload deployed on a different provider or on premises), this is notalways the case; a single virtual machine, for example, is an atomicunit that cannot be split up among providers. In such cases it may notbe possible to take advantage of a reserved instance (or a template),and overall performance considerations govern the deployment.

It should also be noted that reserved instances may include otherrestrictions, e.g., they may be limited to certain geographical regionsor zones and/or require a particular license.

Turning to FIG. 11, according to various embodiments, the supply chain1100 may include two types of entities, namely, Service Entities (SEs),such as a Virtual Machine 1110 or a Container 1120, and Resources, suchas CPU Allocation (CPUAllocation) 1102 and Memory Allocation(MemAllocation) 1101.

In some embodiments, the market may suggest an increase of (scaling up)the Memory Allocation 1101 of the Container 1120, or it may suggest thecreation of another instance (scaling out) of the Container 1120.According to various embodiments, decisions to scale up/down will applyto Resources only, while decisions to scale out will apply to SEs only.

For example, in FIG. 11, the MemAllocation 1101 of the Container 1120may reduce as a result of congestion for resources at the VirtualMachine level. Increased utilization of MemAllocation 1101 of theVirtual Machine 1110 will lead to increased MemAllocation price. Inturn, the increased MemAllocation price increases expenses ofMemAllocation for the Container 1120, leading to a decision to reducethe size of MemAllocation of the Container 1120.

With reference now to a supply chain 1200 shown in FIG. 12, theContainer 1120 consumes directly from a Physical Machine 1210. TheMemAllocation size may also reduce as a result of congestion forresources at the Physical Machine level. Increased utilization ofPhysical Machine MemAllocation will lead to increased MemAllocationprice, which in turn increases expenses for MemAllocation on theContainer 1120, leading to a decision to reduce the size ofMemAllocation of the Container 1120.

Container MemAllocation size may increase as a result of overprovisioned resources at the Virtual Machine level. Decreasedutilization of Virtual Machine CPUAllocation due to a high capacity willlead to decreased CPUAllocation price, which in turn decreases expensesfor CPUAllocation on the Container 1120. If the Container 1120 has highrevenues for CPUAllocation this would lead to a decision to increase thecapacity of CPUAllocation on the Container 1120.

Decisions for both resources and SEs are based on revenues and expensesof these resources. Similarly, expenses and revenues can be set to apredetermined value as desired. For example, the price of MemAllocationcan be set to a minimum value to force higher expenses if attempting tomaintain the size of the MemAllocation of the Container at or below somevalue. This advantageously avoids unnecessary resizing only for thepurpose of having additional MemAllocation. According to otherembodiments, the price of MemAllocation can be set to a maximum value.

The process 1300 shown in FIG. 13 illustrates how a decision is made toscale a resource allocation up or down. The process 1300 firstdetermines if the revenue/expense of a commodity (which may be, forexample, a template rather than a specific resource) is greater than apredetermined value X (decision block 1301). If so, then the capacity ofthe resource is scaled up until the revenues are equal to the expenses(step 1302). If the revenue/expense of the resource is less than apredetermined value Y (decision block 1303), then the resourceallocation is scaled down until the revenues are equal to the expenses(step 1305). Otherwise, if the revenues/expense ratio of the resource iswithin the range defined by the values X and Y (decision blocks 1301 and1303), then the resource allocation is not scaled (step 1304).Advantageously, the values of X and Y provide a mechanism to tune theresponsiveness of the system to increases or decreases in demand. Thevalue of revenues/expenses captures the profitability of the resourceallocation (or the SE). If the ratio is >1, the resource is profitable.If it is <1, it is losing money. In process 1300, X is typically (butnot necessarily) ≥1 and Y is typically (but not necessarily) <1. Statedin another way, an increase in capacity typically is suggested when theresource is profitable, and a decrease when it is operating at a loss.It should be understood, however, that if the consumer is a template,the scale-up decision may be taken in terms based on alternativetemplate offerings as well as, or instead of, individual resourceofferings that may replace or augment the template.

As an additional advantage, decisions capture the entire state of thesystem, and can optimize the system as a whole. Increased utilization ofa resource will lead to increased price for the resource, which in turnincreases expenses for the resource. In some embodiments, the idealprice for scaling the resources provides 70% utilization.

In some embodiments, revenues and expenses can refer to the accumulatedrevenues and expenses over a period of time. Different periods of timecan be used to adjust the decision-making behavior (e.g., aggressiveversus conservative behavior). Short time frames lead to aggressivedecisions, where the system responds very quickly to changes in thesupply and demand anywhere along the supply chain. This can be used, forexample, to respond quickly to congestion for resources and guaranteethe quality of service offered to the entities in the system. Long timeframes dampen the effects of short-term changes in the supply anddemand, and reflect accurately the longer-term trends of the demand andsupply.

A similar decision tree to the one shown in FIG. 13 is depicted in FIG.14, which illustrates an exemplary process 1400 for scaling SEs. Insteadof resizing resources as shown in FIG. 13, the process 1400 creates anew instance of a SE, or suspends the operation of an existing SE,depending on the expenses and revenues of the SE. In particular, theprocess 1400 first determines whether the revenue/expense of a SE isgreater than a predetermined value X (decision block 1401). If so, thena new instance of the SE is created (step 1402). If the revenue/expenseof the SE is less than a predetermined value Y (decision block 1403),then the operation of the SE is suspended (step 1405). Otherwise, if therevenues/expense ratio of the SE is within the range defined by thevalues X and Y (decision blocks 1401 and 1403), then the SE is unchanged(step 1404).

As discussed above, in addition to managing container resources, thesupply-chain principles discussed herein also may be used to manageapplication performance in other virtualization systems. For example, anapplication server requires a certain amount of memory and CPUresources. A database will also require a certain amount of storage. Inorder for the application to perform adequately, the application must beallocated a sufficient amount of each necessary resource. In order forthe infrastructure to be utilized efficiently, the application shouldonly consume what it requires at any given point in time.

Accordingly, with respect to application performance, the supply-chainprinciples discussed in connection with FIGS. 13 and 14 can be used toscale up/down, by adding or removing resources allocated to theapplication, or to scale out, by adding more application components, orsuspend application components. Some examples of application resourcesinclude, without limitation, java heap, thread pools, and connectionpools in an application server, or data space and log space in arelational database. These decisions may be based on a supply chain 1500as shown in FIG. 15, and the revenues and expenses of each of theentities involved in that figure. In particular, the supply chain 1500includes the two types of entities discussed above with reference toFIG. 11. Specifically, the supply chain 1500 illustrates the SEs, suchas the Physical Machine 1210, the Virtual Machine 1120, or anApplication Server 1530, and the Resources, such as Memory (Mem) 1501,Virtual Memory (VMem) 1502, and Heap 1503.

As discussed above, the resources and SEs have expenses and revenues.For example, the revenues of a virtual central processing unit (VCPU)1504 sold by the Virtual Machine 1120 are generated from the ApplicationServer 1530 buying this resource. Expenses of the VCPU 1504 come frompaying to acquire a necessary resource, such as CPU 1505, from theunderlying Physical Machine 1210 hosting the Virtual Machine 1120. In acloud environment, the cloud provider takes the place of the PhysicalMachine 1210 and offers CPUs 1505 and Memory 1501, as a service. Thesales, however, reflect the service packages and templates. For example,while it is possible to add memory to a physical machine is smallincrements, the increments available from a CSP may be less granular;and in the case of a template, one type of resource (e.g., CPUs) may bebundled with another type of resource (e.g., memory) in differentservice packages so that, when comparing templates, the price of oneresource cannot be considered in isolation. It may nonetheless be lessexpensive to procure the template, even if it contains currentlyunneeded resources, than to purchase individual service components on amore granular level. The same is true of reserved instances. Thetechniques described above may be straightforwardly modified to includethis comparison among service-component packages as well as comparisonbetween packages and individual resources offered on-demand. Forexample, suppose a reserved instance is available at a discount thatmakes it 50% cheaper than purchasing the resources on-demand. Even ifthe reserved instance is used only 60% of the time and is otherwiseidle, it is still economically preferable to the on-demand option.

Similarly, a SE has revenues which can be the sum of the revenues of theresources it sells, while its expenses can be the sum of the expenses ofthe resources it buys. As another example, the revenues of the VirtualMachine 1120 can be the sum of the revenues of the VCPU 1504 and theVMem 1502 that it sells to the Application Server 1530 in FIG. 15, whileits expenses are the sum of the expenses to acquire the CPU 1505 and Mem1501 from the Physical Machine 1210.

Revenues and expenses can depend on the prices of resources, which inturn can be a function of supply, e.g., attributes of the resource suchas its capacity, as well as the demand—how much of the capacity iscurrently utilized by resources or SEs consuming this resource. In oneembodiment, price is a function of the utilization (U) of the resource,and depends on it through the formula 1/((1−U)².

For example, an application server requires java heap in order toprocess transactions. This java heap is allocated from the underlyingvirtual machine's virtual memory allocation. In the event that thedemand for java heap is very high (e.g., generating revenue for theapplication server), and the price of virtual memory from the virtualserver (e.g., determined by the combination of supply and demand) issufficiently low, then the application server will be able to buy morevirtual memory from the virtual server and allocate additional javaheap. In the event that the demand for java heap is low and the price ofvirtual memory is high then the application server will decrease itsallocation of java heap and return virtual memory to the virtual machineto be used by other applications.

In some embodiments, the buyer can be assigned a budget for purchasingthe resources. Decisions for both resources and SEs are based on therevenues and expenses of these resources. Similarly, expenses andrevenues can be set to a predetermined value as desired. For example,the price of VMem can be set to a minimum value to force higher expensesif attempting to maintain the size of the Heap at or below some value.This advantageously avoids unnecessary resizing only for the purpose ofhaving additional VMem. According to other embodiments, the price ofVMem can be set to a maximum value.

In some embodiments, the market may suggest to increase (scale up) theHeap size of an Application Server, or it may suggest to create anotherinstance (scale out) of the Application Server. These decisions can bebased on the process 1300 for resizing resources and process 1400 forscaling SEs as discussed above. Also as discussed above, revenues andexpenses can refer to the accumulated revenues and expenses over aperiod of time and different periods of time can be used to adjust thedecision-making behavior (e.g., aggressive versus conservativebehavior). For example, longer periods of time can be used to anticipatefuture needs for extra application servers based on steadily increasingrevenues that reflect an increase in demand. Conversely, a longer termdecrease in revenues indicates that the steady state operation of asystem may not require a particular SE.

The use of supply chain economic principles and other principlesexplained above serve several purposes and provide several potentialbenefits, both expressly numerated and otherwise. For example, theseprinciples can be used to provide a common software framework andabstractions to unify and automate the management of container systems.More specifically, they can be used to optimize or improve theallocation of IT resources (such as I/O resources or software licenses)to best process applications workloads according to their businessvalue. The principles of supply chain economics can also be used tobalance workloads to minimize disruptive operating conditions, such asI/O congestion, and to reduce resource waste by terminating orswitching-off underutilized resources. These principles can also be usedto empower business units to monitor and control the delivery of SLAs totheir applications, as well as the ROI of individual elements and theoverall container system. In addition, for example, these principles canbe used to handle the management of virtual resources in a multi-cloud(or multi-domain) system.

Additionally and/or alternatively, the management of resources incontainer systems and conventional virtualization systems can includenot only supply-chain based methods, but also regulation of access tothe resources. FIG. 16 illustrates an exemplary system 1600 forregulating access of consumers 1610 (e.g., electronic applications) toresources and services (e.g., storage). In one embodiment, thisregulation occurs through the use of access permits (not shown) that theconsumer 1610 acquires from an intermediate entity—an Action Manager(AM) 1620—prior to accessing the resource or service. As shown in FIG.16, the AM 1620 regulates access to a provider 1630 of the resource orservice. For example, regulating access includes controlling the numberof concurrent accesses, and/or the rate at which consumers 1610 accessthe resource, as desired.

In some embodiments, there is one type of permit per provider 1630.According to various embodiments, the AM 1620 can sell multiple types ofaction permits, regulating access to a number of resources. Each permitcan be associated with a predetermined price. Additionally oralternatively, this price can be dynamically adjusted taking intoconsideration the availability of permits possessed by the AM 1620.Permits sold by the AM 1620 can create both revenues and expenses forthe AM 1620. The revenues come from the price the consumer 1610 has topay to the AM 1620 to buy the permit. The expenses come from the pricethe AM 1620 has to pay to the resource provider 1630 for the right tosell these permits. For example, the AM 1620 may need to pay forInput/output Operations Per Second (IOPS) offered by a storagecontroller in order to allow access to the consumer 1610.

In some embodiments, the price that the AM 1620 pays for the right tosell these permits is determined by the provider 1630 based on one ormore of the following parameters: the capacity and the percentage theprovider 1630 wishes to make available to the consumers 1610; thecurrent load of the provider 1630; and the rate at which the provider1630 wishes its resources to be accessed.

The AM 1620 can dynamically adjust the number of permits it possesses atany time, depending on its revenues and its expenses. For example, ifthe AM 1620 is profitable (e.g., the charges based on price it isselling the permits to the consumer 1610 is higher than the chargesbased on price it pays to the provider 1630 for the right to sell thesepermits), the AM 1620 can consider increasing the number of permits itsells. Alternatively, if the AM 1620 is losing money, the AM 1620 canconsider decreasing the number of permits it is selling.

Advantageously, the AM 1620 can be used to avoid I/O congestion instorage controllers when several VMs request to execute heavy-storageapplications (e.g., VM Reboots, Antivirus database updates, OS Updates,and so on) at the same time. In one embodiment, the AM 1620 limits thenumber of concurrent consumers that can access the provider 1630. It maylimit access across types of applications or within each type ofapplication. For example, permits can be priced and provided for allanti-virus, OS updates, etc. separately, or all of them may beconstrained by the same permits. In this example, the provider 1630 isthe storage controller, while the consumer 1610 is the applicationperforming the heavy-storage task. For instance, the application can beperforming an anti-virus update on the virtual machine.

Turning to FIG. 17, the consumer 1610 (e.g., an application) sends theAM 1620 a request 1601 to acquire the appropriate number of permits(e.g., 5) for the provider 1630 (e.g., a storage controller) of thestorage associated with the VM. It will be understood that, althoughreference is made to a storage controller with respect to FIG. 17,according to various embodiments, other types of providers and resourcesare managed using similar principles and permits. After a request 1601has been received, the AM 1620 subsequently determines 1602 if therequest includes a sufficient budget, and if the AM 1620 has enoughpermits to satisfy the request 1601. If so, the AM 1620 replies to theconsumer 1610 with the appropriate permits and charges. After buying thepermits, the consumer 1610 accesses 1602 the storage through theprovider 1630 and performs the update. After completing the update, theconsumer 1610 releases 1604 the permits such that the AM 1620 canre-sell them. The AM pays 1605 the provider 1630 for the use of thepermits it is selling. According to various embodiments, payment for theuse of permits can occur before, after, or simultaneously with storageaccess.

In an alternative embodiment, the number of concurrent accesses to aresource may vary. For example, the AM 1620 adjusts the number ofpermits it is selling, to reflect the ability of the provider 1630 tosatisfy concurrent requests by consumers 1610. For example, when the AM1620 pays the provider 1630 for the use of the permit, the AM 1620adjusts the number of the permits it sells based on how profitable itis. If demand for permits for a specific provider 1630 is high, the AM1620 raises the prices for this permit, advantageously increasingrevenues.

To become even more profitable, the AM 1620 can request the right tosell more permits from the provider 1630. If the provider 1630 agrees,the provider 1630 raises the price the AM 1620 has to pay for theserights. As the demand increases, the provider 1630 continues to increasethe price it charges the AM 1620. At a threshold price, the AM 1620 canno longer make a profit, and the AM 1620 does not request any furtherincrease in the number of rights it can sell. Similarly, the number ofpermits sold by the AM 1620 can decrease as a result of reduced demandby consumers 1610, or increased prices by the provider 1630.

In yet another embodiment, the AM 1620 controls rate of concurrentaccesses to a particular resource. For example, the AM 1620 limits therate at which the applications are accessing the storage controller toperform the heavy-storage tasks. In this case, once the applicationreleases the permit, and until the predetermined period of time haselapsed, the AM 1620 cannot resell this permit. The storage controllercan charge the AM 1620 a very small amount for the right to sell a firstpredetermined number of permits within a period of time, and thenincrease the price to infinity for permits beyond the firstpredetermined number in this period. The consumer request to access oneor more permits may be made, for example, directly to the resource orservice provider. The AM 1620 may control the total number and/or therate at which a group of consumers accesses a group of resources.

Another aspect discussed above formulates and evaluates the option tomove the consumer to a new provider. “Formulating” includes theattributes taken into account when considering the option to move to thenew provider. The cost of moving can be part of the comparison betweentwo different alternatives (e.g., keeping a VM in an existinginfrastructure or moving the VM to an external cloud provider). Cost canbe expressed in actual currency or any unit suitable for the comparison.For example, moving time can be expressed in a real value thatquantifies the cost of the VM downtime. In contrast, if there is astrict limit on acceptable downtime, the cost of moving the VM can beexpressed in terms of time.

“Evaluating” includes making the decision (e.g., initiating an actionbased on the decision) and determining the right time to take theaction. Compared to other economics-based decision-making systems, oneembodiment described herein postpones the decision for the future,advantageously waiting for a sufficient amount of time until thedecisionmaker is convinced that the decision is the right one.Evaluating may occur among on-demand offerings, among template orreserved-instance offerings, or among both, in which case a qualifyingtemplate or reserved instance will have all required components, and mayhave additional components. The additional components may receive noweight in the comparison, i.e., the cost of the bundled offering iscompared strictly against procurement of the required componentsindividually, from on-demand service offerings. Alternatively, theadditional components may have a value based on, for example, predictedneed. If there is some measurable or assignable chance that theadditional components will be needed during the CSP contract period,they may be assigned a value that reflects this probability and thisvalue may be subtracted from the cost of the bundled services, making itmore economically attractive.

For example, a virtualization system is considering taking an action Awith the cost of taking this action represented as C(A). If the actionis taken, the savings over time is S(t). The decision to take the actionat the time to when the savings would have exceeded the cost of theaction is represented by the equation S(tA)≥C(A).

In one embodiment, with reference to FIG. 18, a virtualization system1800 controls moves of VMs 1810 between different storage (or resource,e.g., CSP) providers 1820 to avoid frequent moves of VMs 1810 betweendifferent storage providers 1820 in a data center (DC) 1830 or acrossdifferent data centers. For example, the VM 1810 is evaluating a move toone or more service providers 1820, such as storage providers SP₁, SP₂,. . . SP_(N). Although storage providers 1820 are used herein as anexample, it will be understood that the concepts disclosed herein can beapplied to other types of service or resource providers, including CSPs.

In some embodiments, the cost C(Ai) of moving to provider i is set to avalue proportional to the size of the data to be moved from the currentSP to SP_(i), multiplied by a factor P_(i) that captures the ‘proximity’of the current SP to SP_(i). For example, if the current and the futureSPs are in the same data center 1830, P_(i) could be set to 1, whereasif they are in different data centers 1830, it could be set to 10, tocapture that it is more expensive to move across data centers 1830 thanto move within the same data center 1830.

The consumer periodically checks the prices at each provider i,including the current provider, calculates the saving for this periodand adds them to the savings from the previous periods. The price of thenew provider for the current period may be higher than that of thecurrent provider, and as a result the savings for this period will benegative and will decrease the total savings from previous rounds. Whenthe accumulated savings exceed the cost C(A_(i)), the VM 1810 decides tomove SP_(i).

In an alternative embodiment, when the consumer considers moving to anew provider, the current provider gives the consumer some credit (e.g.,equal to C(A)) to convince the consumer to stay. The consumer acceptsthe credit, and periodically checks the price of the new provider. Ifthe price is cheaper, the consumer can use this credit to subsidize anyloss of not having moved there. If it is more expensive, the consumeradds her gain to the credit. If the consumer runs out of credit, thenthe consumer can decide to move. Reserved instances can operate in thesame manner, i.e., the decision whether to move will reflect anypaid-for but as-yet-unused portion of the reserved instance, as well asany promotion associated therewith for renewals or future service.

Advantageously, the system accounts for the fact that a decision thatlooks good now may not be good in the future. For example, in anon-demand procurement, a consumer that buys bandwidth from a networkprovider may see a cheaper price offered right now by a new provider.However, the new provider may change the price an hour later, and thisnew price may be higher than the price of the current provider an hourlater.

Additionally, the system accounts for the actual behavior of otherusers. Assume a VM is interested in the latency of accessing data storedon a disk, and a decision is made to move its data from the current to anew disk that currently has lower latency. For large amounts of data,the move could take hours to complete. While the move takes place, otherconsumers who also see a slightly reduced latency move to the same newprovider—effectively increasing the latency for everyone, and making ita bad decision.

Furthermore, the amount of time it takes to determine that the decisionmay be good is related to the cost of performing the action. Therefore,expensive decisions are carefully validated over longer periods thancheaper decisions, ensuring that undertaking the cost of the action willpay off in the future. Advantageously, the systems and methods aboveminimize bad decisions and decisions that would frequently alternatebetween the current and the new provider.

In some embodiments, the economically based cost analysis disclosedherein can be used to migrate workloads among multiple providers in acloud environment. For example, one or more private clouds or datacenters, one or more public clouds or data centers, or a combination ofthe two, each can sell a commodity referred to as “cost” to a virtualmachine or container. Therefore, when a virtual machine, container, orother entity determines whether to migrate workloads in a cloudenvironment, the entity considers the priced charged by the cloudprovider to migrate there. Stated in another way, the real cost ofrunning demand on a cloud service provider or platform can beincorporated into any decision to migrate a workload to a public cloudbased on the price of the provider as well as utilization.

With reference to FIG. 19, a virtualization system 1900 controls movesof VMs 1810 between cloud providers 1930 and private data centers 1940to avoid overall congestion of the private data centers 1940.Advantageously, the more utilized a selected private data center is, themore lucrative the public providers (e.g., cloud providers 1930) become.For example, the VM 1810 may evaluate a potential move to one or moreservice providers (e.g., cloud providers 1930 and private data centers1940), such as storage providers SP₁, SP₂, . . . SP_(N). Although thecloud providers 1930 and private data centers 1940 are used herein as anexample, it will be understood that the concepts disclosed herein can beapplied to other types of service or resource providers.

In some embodiments, the cost C(Ai) of moving to provider i is set toC(Ai)=[1/(1−X)]², where X=(Total_spent+On_Demand_price)/(Total_Budget).For example, the total_spent is the total amount spent by the consumerper hour/week on all service providers, the On_Demand_price is theon-demand (as opposed to bundled) price to run the load now on aspecific service provider, and the Total_Budget is the consumer's totalbudget for running the load in a particular service provider perhour/week. For the cloud providers 1930, the used value of the cost isthe cost of running a current workload on a selected SP. The capacity ofthe cost is the budget specified by the consumer for running theworkload in the cloud provider 1930. Some commodities can have infinitecapacity, making the cost the dominant pricing commodity.

For providers in the private data center 1940, the used value of cost isthe dollar value of running a current workload on the private datacenter 1940. The capacity of cost can be infinite, making othercommodities dominant. For a hybrid provider found both in the privatedata center 1940 and the cloud provider 1930, the cost of a move is alsoconsidered, both internal and across cloud providers 1930. The costC(Ai) of moving to provider i remains the same, therefore, getting moreexpensive to migrate when the cost moves closer to an allocated budgetfor running on the cloud.

Accordingly, the more utilized a private data center 1940 is, the morelucrative the cloud provider 1930 becomes. Spot price fluctuations inindividual or bundled cloud services can directly (or indirectly) affectany decisions to migrate. If there is no congestion in a private datacenter 1940, the cost of a migration prevents demand from moving to acloud provider 1930. Once the private data center 1940 becomes congestedto a predetermined level, the cost of migration to a cloud provider 1930can be desirable. Once the budget allocated for running on the cloudprovider 1930 is close to being met, the prices of running on a cloudprovider 1930 become higher causing demand to remain on a private datacenter 1940 and a migration decision can be made.

In various embodiments, the consumer periodically checks the prices atthe current and each provider i, calculates the saving for this periodand adds them to the savings from the previous periods. The price of thenew provider for the current period may be higher than that of thecurrent provider, and as a result the savings for this period will benegative and will decrease the total savings from previous rounds. Forexample, the moment the savings up to now exceed the cost C(A_(i)), theVM 1810 will decide to move to SP_(i).

In some embodiments, the price of running a workload on a private datacenter 1940 can be unchanged (e.g., price=1). Alternatively, forexample, the price can represent the real cost of running workloads onthe private data center 1940 as a combination of cost of facilities,capital amortization, and/or operations cost. The price may alsorepresent other combinations of these and/or other types of costs.

The pricing can further extend to differentiate between serviceproviders based on performance as well as cost. This adds a tradeoffbetween price and performance. For example, if SP₁ has better CPUperformance than SP_(N), the price of CPU on a SP₁ may be effectivelycheaper, resulting in workloads consuming a significant portion of CPUpreferring to run on a SP₁ even though the cost of running on a SP_(N)is cheaper. Stated differently, superior performance of a cloud providercan render a nominally more expensive resource as a more cost effectiveoption. For example, if a first provider has superior CPU performancecompared to a second provider, it is possible that usage of the firstprovider, even at a higher nominal cost for CPU usage, results inoverall cost savings compared to usage of the second provider, with alower nominal cost (but also lower performance) for CPU usage. In thisscenario, for example, the systems, methods and apparatus can encouragethe migration of consumers to the first provider.

In some embodiments, once a certain demand is deployed or migrated to apublic cloud, the entity will continue to shop for alternativeproviders, both for on-demand and reserved-instance offerings, andrelocate if the cost is too high. Also, an entity may be configured tostrongly or exclusive favor template-based service procurement. Forexample, if a given application requires a minimum amount of CPU andmemory capacity, the entity may be a “template entity” that seeks topurchase cloud services on a template basis rather than as individualresources, since template procurement conveniently ensures bothsatisfaction of minimum resource requirements and retention of theseminimum requirements with a single CSP; if a migration decision is made,it applies to the full template of services and will occur only ifjustified on that basis.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a tangible program carrier forexecution by, or to control the operation of, data processing apparatus.The tangible program carrier can be computer readable medium, such as amachine-readable storage device, a machine-readable storage substrate, amemory device, or a combination of one or more of them.

The terms “data processing apparatus” “data processor”, or “processingdevice” encompasses all apparatus, devices, and machines for processingdata, including by way of example a programmable processor, a computer,or multiple processors or computers. The apparatus can include, inaddition to hardware, code that creates an execution environment for thecomputer program in question, e.g., code that constitutes processorfirmware, a protocol stack, a database management system, an operatingsystem, or a combination of one or more of them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio or video player, a game console, a GlobalPositioning System (GPS) receiver, to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described is this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a LAN and a wide area network (“WAN”),e.g., the Internet.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter described in thisspecification have been described. Other embodiments are within thescope of the principles discussed herein. For example, the actionsrecited in the drawings can be performed in a different order and stillachieve desirable results. As one example, the systems, apparatus andmethods depicted in the accompanying drawings do not necessarily requirethe particular configurations or order of processes shown (or sequentialorder) to achieve desirable results. In certain implementations,multitasking and parallel processing may be advantageous.

1. A computer-implemented method, comprising: determining, by a consumermanager running on a data processor in a computer system, a set ofresources for running a workload required by a component of the computersystem, the resources including a needed quantity of one or more ofCPUs, memory, databases, network bandwidth, and/or input-outputcapacity; identifying, by the consumer manager, a first cloud-basedresource offering by a first provider including the needed quantity ofresources individually on an on-demand basis reflecting a utilizationlevel of the resources and determining a cost for provisioning theneeded quantity of resources to the computer system on the on-demandbasis; identifying, by the consumer manager, a second cloud-basedresource offering by a second provider different from the firstcloud-based resource offering and consisting of a reserved instance, thereserved instance specifying a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity fully utilized for a fixed time period andcorresponding to or exceeding the needed quantity of resources;determining a cost charged for the reserved instance; selecting toutilize the reserved instance to run the workload based at least in parton a comparison of the determined cost for running the workload on thefirst provider given the utilization level with the determined cost ofthe reserved instance.
 2. The method of claim 1, further comprising thesteps of: determining, after passage of a predetermined period of timeno greater than the fixed time period associated with the reservedinstance, a cost for running the workload on a third cloud serviceprovider; determining a cost of moving the workload to the third cloudservice provider; computing a utilization value for running the workloadon the third provider based at least in part on the determined cost forrunning the workload on the third provider, the determined cost ofmoving the workload to the third provider, and a need by one or moreother components of the computer system for the remainder of thereserved instance; and moving the workload to the third provider basedat least in part on the utilization value after the utilization valuehas surpassed a predetermined value.
 3. The method of claim 1, whereinthe first and second second cloud-based resource offerings are offeredby different cloud providers.
 4. The method of claim 1, wherein thefirst and second second cloud-based resource offerings are offered bythe same cloud provider.
 5. The computer-implemented method of claim 1,wherein the reserved instance exceeds the needed quantity of resourcesand the computing step accounts for any restriction on moving an excessportion of the reserved instance to another component of the computersystem.
 6. The computer-implemented method of claim 1, wherein the costof running the workload on the second provider is based at least in parton the utilization of one or more resources of the first provider. 7.The computer-implemented method of claim 1, wherein the cost of runningthe workload on the second provider is based at least in part on adetermined performance characteristic of the first or second provider.8. The computer-implemented method of claim 1, wherein the cost ofrunning the workload on the third provider is a dynamic, on-demand pricebased on one or more characteristics of the computer system.
 9. Thecomputer-implemented method of claim 1, wherein the determined cost forrunning the workload on the second provider is based at least in part onone or more of an actual or anticipated environmental impact, acontractual clause associated with the reserved instance, a preferredvendor status, a quality of service (QoS) requirement, or a complianceor regulatory requirement.
 10. The computer-implemented method of claim1, wherein the determined cost for running the workload on the secondprovider is based on one or more of: a cost of facilities, a capitalamortization, or an operations cost.
 11. The computer-implemented methodof claim 1, further comprising exchanging virtual currency units usedfor running the workload to a government-backed currency.
 12. Acomputer-implemented method, comprising: determining, by a consumermanager running on a data processor in a computer system, a set ofresources for running a workload required by a component of the computersystem, the resources including a needed quantity of one or more ofCPUs, memory, databases, network bandwidth, and/or input-outputcapacity; determining, by the consumer manager, a cost in currency unitsfor running the workload on a first cloud service provider as a firstreserved instance specifying a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity fully utilized for a fixed time period, the firstreserved instance corresponding to or exceeding the needed quantity ofresources; determining, by the consumer manager, a cost in currencyunits for running the workload on a second cloud service provider as asecond reserved instance specifying a quantity of one or more resourcesselected from CPUs, memory, databases, network bandwidth, and/orinput-output capacity fully utilized for a fixed time period, the secondreserved instance being different from the first reserved instance butcorresponding to or exceeding the needed quantity of resources;selecting to run the workload on either the first or second reservedinstance based at least in part on a comparison of the determined costs;determining a cost for running the workload on a third provider in thecomputer system, wherein the third provider is an in-house provider,based on a historical resource utilization of the workload; determininga cost of moving the workload to the third provider in the computersystem; computing a cost for running the workload on the third providerbased at least in part on the determined cost for running the workloadon the third provider and the determined cost of moving the workload tothe third provider; and moving the workload to the third provider basedon the computed cost.
 13. The computer-implemented method of claim 12,wherein the first cloud service provider is different from the secondcloud service provider.
 14. The computer-implemented method of claim 12,wherein the first and second cloud service providers are the same cloudservice provider.
 15. The computer-implemented method of claim 12,wherein the selected reserved instance exceeds the needed quantity ofresources and the computing step accounts for any restriction on movingan excess portion of the selected reserved instance to another componentof the computer system.
 16. The computer-implemented method of claim 9,wherein the cost of running the workload on the third provider is adynamic, on-demand cost based on one or more characteristics of thecomputer system.
 17. A computer system for managing allocation ofworkloads, comprising a consumer manager configured to: determine a setof resources for running a workload required by a component of thecomputer system, the resources including a needed quantity of one ormore of CPUs, memory, databases, network bandwidth, and/or input-outputcapacity; identify a first cloud service provider offering the neededquantity of resources individually on an on-demand basis reflecting autilization level of the resources and determining a cost charged by thefirst cloud service provider for provisioning the needed quantity ofresources to the computer system; identify a second cloud serviceprovider different from the first cloud service provider and offering afirst reserved instance, the first reserved instance specifying aquantity of one or more resources selected from CPUs, memory, databases,network bandwidth, and/or input-output capacity fully utilized for afixed time period and corresponding to or exceeding the needed quantityof resources; determine a cost charged by the second cloud serviceprovider for the reserved instance; select to purchase the reservedinstance to run the workload on the second provider based at least inpart on a comparison of the determined cost for running the workload onthe first provider with the determined cost for running the workload onthe second provider; determine, after passage of a predetermined periodof time no greater than the fixed time period associated with the firstreserved instance, a cost for running the workload on a third cloudservice provider as a second reserved instance specifying a quantity ofone or more resources selected from CPUs, memory, databases, networkbandwidth, and/or input-output capacity fully utilized for a fixed timeperiod and exceeding the needed quantity of resources; determine a costof moving the workload to the third cloud service provider; compute autilization value for running the workload on the third provider basedat least in part on the determined cost for running the workload on thethird provider, the determined cost of moving the workload to the thirdprovider, and a need by one or more other components of the computersystem for the resources of the second reserved instance exceeding theneeded quantity of resources; and cause the workload to be moved to thethird provider based at least in part on the utilization value after theutilization value has surpassed a predetermined value.